HCAP-1396: Create a seeding script for other tables

rhattie-ey commented 1 year ago

Original ticket: HCAP-1396

Work completed

`feed-data` rewrite

Overhauled the entire script, rewriting it from the ground up.

Features:

Encompasses all tables needed for testing
Simpler
More robust error handling
- Specific logic to handle and report foreign key errors
More human-friendly console output
Fully compliant with strict TS
Generalized enough to work for any table in the database
Importable for use in wrapper scripts
Has a --skip-missing-tables flag to allow population of incomplete databases (such as for server tests)

Test data generation

Jupyter notebook used for generation

Created new test data CSVs for the following tables:

cohort_participants
cohorts
participant_post_hire_status
participants
participants_status
phases
post_secondary_institutions
return_of_service_status
site_phase_allocation

Misc

Updated docs
Removed the now unneeded copy-sample-data command
Updated scripts, makefiles, and actions to use new scripts
Deleted old test data from server/script/csv and server/test-data, including all XLSX files other than those in server/tests/mock
Flagged code that is now deprecated due to this PR
Updated Node.js to LTS release 18.15.0
Updated tests to work with new data
Updated GitHub actions to use only feed-data for seeding

Next steps

fw-noel commented 1 year ago

I have one test to try and resolve, but in the meantime, couple of comments Some tests have heavy reliance on specific ids which caused a few of the breaks. In my commit, you'll see that 26, 27, and 28 were important ones. We'll want to improve this in the future by just moving the necessary participants through in the tests themselves.

The other issues are that all the csvs start with 0 ids, they should start at 1

Also, massivejs insert doesn't add to the sequence of a table, so there needs to be a way to ensure that we can save in the future into tables with auto generated pkeys. I added