Create Postgres dump of just the ~74 timeseries datasets required by Causeex

[saggu]

To do:

• Use Datamart URL column of Causeex timeseries spreadsheet https://drive.google.com/drive/u/2/folders/0ANVWShV1oMlpUk9PVA to fine the dataset id and variable id.
• Write SQL query to extract the subset of database

[kyao]

Need to check with them if indicator name should be the variable id

[saggu]

Things to do to achieve this ,

1. Load all data from the full database backup.
2. Delete metadata and data for variables which are not needed for CausX (@kyao knows these required variables)
3. Create a sql back for the remaining data
4. In a new branch, create a docker compose with this CausX database backup

[zmbq]

I do not have permission to view the spreadsheet for some reason, so I cannot write the script now

[szeke]

Is this resolved? @zmbq

[zmbq]

Uploaded to the CausX shared drive.

The development branch has a handy script in script/clean-for-causx.py, which accepts the excel spreadsheet as an argument, and cleans Datamart - deleting all the other variables. It also removes empty datasets. The script has a few handy arguments (please use --dry-run the first time you try it).

Running it on the full Datamart takes a couple of hours, since it deletes a lot of variables.