Closed gracebrownecodes closed 7 years ago
Yes, load includes adding indexes and FKs.
How about a document that describes and provides examples for all of the steps, including ones we are not actually demo'ing. This could be the basis of the "PEDSnet database management" documentation.
Re: the load
command, there is an open PR that should be merged since it was used in production. I just made another ticket with some of the known TODO items for the load
command.
Re: the demo environment, if we use db01, we can either create a special demo cluster or we can let Ritu et al know we are co-opting pedsnet_dcc_v22
temporarily and rename the original v22 database.
Steps that aren't demo'd that could appear in our document:
So my script is something along the lines of:
Steps for initial loading
Expand the data package into an arbitrarily named subdirectory (here: demo_v22
, but perhaps best named after the data cycle)
infomodels expand -o demo_v22 demo_v22.tar.gz
Validate the data package:
infomodels validate --modelv 2.2.0 demo_v22
Load the data for the appropriate site (here: chop
, and we are using the special database instance on port 5433)
infomodels --model pedsnet-core --modelv 2.2.0 load -s chop_pedsnet,vocabulary -d postgresql://loading_user@localhost:5433/pedsnet_dcc_v22?sslmode=disable demo_v22
Things to note: currently the --modelv switch is required, even though the tool should be able to derive the model version from metadata.csv
.
And then I'll pick up with my script, which goes something like.
pedsnetdcc check_fact_relationship -s chop_pedsnet postgresql://loading_user@reslnpedsndb01.research.chop.edu:5433/pedsnet_dcc_v22
pedsnetdcc sync_observation_period -s chop_pedsnet postgresql://loading_user@reslnpedsndb01.research.chop.edu:5433/pedsnet_dcc_v22
<site>_id_maps.<domain>_ids
).0
value in the DCC last ID table for each domain and insert the value from the previous dataset (dcc_ids.dcc_<domain>_id
).pedsnetdcc transform -s chop_pedsnet,chop_id_maps,dcc_ids,vocabulary --site chop -v 2.2.0 postgresql://loading_user@reslnpedsndb01.research.chop.edu:5433/pedsnet_dcc_v22
pedsnetdcc merge -v 2.2.0 postgresql://loading_user@reslnpedsndb01.research.chop.edu:5433/pedsnet_dcc_v22
search_path
by default based on the site nameDone.
The setup is:
The demo is:
infomodels expand
a data archiveinfomodels validate
the data directoryinfomodels load
the data directory (this adds the foreign keys, right?)pedsnetdcc check_fact_relationship
pedsnetdcc sync_observation_period
pedsnetdcc transform
pedsnetdcc merge
Which means the needs are:
pedsnetdcc
CLIpedsnetdcc
CLIWhat did I miss?