This is the bare bones MVP of sourcing column metadata from our DB.
Here's what this PR does:
Remove crufty metadata yaml files that were mistakenly committed in early days of the data model work
Deprecate the use of the Google sheet that tracks column data and instead store it in JSON files.
Updates all of the python scripts that used the google sheet to use the new JSON files
Adds a _column_metadata table to the database which is intended to be the authoritative source for column information that can be consumed by ETLs and the VZE. This new _column_metadata table serves a different purpose than the JSON files that are replacing the Google sheet. Those JSON files are only needed to generate the data model migrations—they can be discarded after we launch the new data model.
Updates the cris_import.py script to use the _column_metadata table
Adds a new python script, 70_make_column_metadata_table.py, which can be used to generate the _column_metadata seeds, which are parsed out of the column JSON files. This will allow us to continue to keep the metadata seeds in sync with other changes we make to the db schema up until we launch
This PR does not make use of the _column_metadata table anywhere in the VZE. I will open follow up tasks for that.
Associated issue
This is the bare bones MVP of sourcing column metadata from our DB.
Here's what this PR does:
_column_metadata
table to the database which is intended to be the authoritative source for column information that can be consumed by ETLs and the VZE. This new_column_metadata
table serves a different purpose than the JSON files that are replacing the Google sheet. Those JSON files are only needed to generate the data model migrations—they can be discarded after we launch the new data model.cris_import.py
script to use the_column_metadata
table70_make_column_metadata_table.py
, which can be used to generate the_column_metadata
seeds, which are parsed out of the column JSON files. This will allow us to continue to keep the metadata seeds in sync with other changes we make to the db schema up until we launchThis PR does not make use of the
_column_metadata
table anywhere in the VZE. I will open follow up tasks for that.Testing