Feat: user dynamic_data sync and import

chrismclarke commented 2 months ago

PR Checklist

[x] PR title descriptive (can be used in release notes)

Description

Main Changes

Add debug page to view current user contact fields and dynamic data
Add migration to include dynamic_data column with server db
Update app sync and import methods to include dynamic_data

Additional Changes

Add a debounce to reduce the frequency user dynamic data is written to disk (max once per 500ms). This seemed a sensible optimisation given that import operations happen 1 row and a time, so could easily trigger a significant number of operations which may block the main thread and make the app hang
Remove API /tables endpoint (as suggested in #2277)

NOTE Production changes will only take effect once merged and server code updated (follow-up task)

Review Notes

In order to populate dynamic data recommend viewing http://localhost:4200/template/comp_data_items

User debug page available at http://localhost:4200/user

Sync testing requires a running development server environment (API + DB)

Dev Notes

We don't currently have a set of spec tests to add db persist debounce test to, and feels beyond scope of this PR to create all from scratch.

The persist operation has been tested locally by adding console logs to src\app\shared\services\dynamic-data\adapters\persistedMemory.ts public persistStateToDB and private handleStatePersist methods, and checking when importing user data that the the handleStatePersist method is only invoked once despite multiple calls to persistStateToDB.

Author Notes

The /user debug page hasn't been included in the debug content app menu (but could in the future)

The list of dynamic_data is formatted as a single column in the existing app_users table, nested first by flow_type (for now all are data_lists) and then by flow_name. E.g. for the synced user the dynamic_data column entry is formatted as:

{
  "data_list": {
    "comp_data_items_list": {
      "id_1": {
        "id": "id_1",
        "label": "Task 1",
        "completed": true
      },
      "id_2": {
        "id": "id_2",
        "label": "Task 2",
        "completed": false
      },
      "id_3": {
        "id": "id_3",
        "label": "Task 3",
        "completed": false
      }
    }
  }
}

Git Issues

Closes #2246

Screenshots/Videos

User Sync and Import demo Create dynamic data entries using the comp_data_items template, and view available user data on new user debug page. Sync user data to db (running locally), and then clear cache to generate a new user profile. Import initial user to see contact fields and dynamic data restored.

Screenity video - Apr 19, 2024.webm

Example DB User entry now includes dynamic_data column with user data

github-actions[bot] commented 2 months ago

Visit the preview URL for this PR (updated for commit eca482c):

https://plh-teens-app1--pr2296-feat-user-data-sync-cgukg2ql.web.app

_{(expires Fri, 17 May 2024 16:12:21 GMT)}

_{🔥 via Firebase Hosting GitHub Action 🌎}

_{Sign: e4c0bab6b08dd290fbf002fd6e07987fa4b5fce1}

chrismclarke commented 2 months ago

I first tried testing locally, but struggled to get my local API set up. I then tested on the web deployment preview (but I'm not sure whether it's expected to work there!)

The /user page seems to work correctly (I could import a user succesfully), but importing the dynamic data did not work. I also couldn't find the dynamic data on the server database. What table should this sit in?

Unfortunately it won't work on web deployment until I've deployed the changes there, so for now the only way to test is to have a database running locally. When running locally (or after future deployment) the data will sit in the same app_users table as the contact_fields

N.B. I was seeing a console error

error-handler.service.ts:65 ERROR TypeError: Cannot read properties of undefined (reading 'persistStateToDB')

That's weird, was this on the user page or somewhere else? Do you know if it appeared on first load or after you clicked to sync/import?

chrismclarke commented 2 months ago

Also, I get the same error that Esmee documented above when viewing the hosted PR preview, on the initial load of the app home page. I haven't seen it locally though. Does the writeCache (PersistedMemoryAdapter) need some kind of initialisation method that can be awaited by the dynamic data service before calling its methods? Or else perhaps the services that call dynamicDataService.getState() (user meta and server) should be awaiting dynamicDataService.ready() before doing so?

Most likely, I've just gone through all the services that import the dynamicDataService and added 33ab0d4 to try address any potential issues - I think error should be resolved now (not seeing the console error on preview link) @esmeetewinkel - let me know if you still see

Otherwise thanks for testing everything else and the review @jfmcquade , hopefully should now be good to go (?)

chrismclarke commented 2 months ago

I'm still seeing the error on the web preview (which I checked has been updated for the last commit) when loading the /user page

Thanks @esmeetewinkel - I had only been checking the home page, not the user page but now I can see the same error. It related to the server sync service which didn't properly wait for the dynamic data service to be ready before trying to read the user tables from it. Fixed with aedfbcd

IDEMSInternational / open-app-builder