Socrata export ETL

[johnclary]

Associated issues

• https://github.com/cityofaustin/atd-data-tech/issues/18149

This is the new Socrata export! The diff is not as bad as it looks—I had to renumber a bunch of migrations.

These follow-up tasks will be addressed in separate issues/PRs:

• airflow
• docker image CI
• prepare/coordinate updates to the production datasets to make them match the changes made here

Testing

You can run this using a local db, but we're going to use the test instance cluster since it is already loaded up with 10 years of data.

Setup

1. In our password store, locate the item called Env file for the Vision Zero new data model Socrata export ETL and save it in the root of this ETL directory as .env.

2. Build and run the Socrata export. This may take up to 20 minutes.

# from ./atd-etl/socrata_export
$ docker compose run socrata_export
$ ./socrata_export.py --crashes --people

3. Nice! Start your local VZV and inspect the summary page and map

# from ./atd-vzv
$ nvm use
$ npm run start

4. Test out the helper script, which logs the differences between the staging datasets and production datasets, which will need to be brought into sync with the staging datasets when we launch.

# from ./atd-etl/socrata_export
$ docker compose run socrata_export
$ ./_compare_column_metadata.py

[roseeichelmann]

i think the test instructions need to be updated to say $ ./socrata_export.py --crashes --people?

also could update test instructions to say to run the hasura migrations/metadata in this branch

[chiaberry]

I didn't run it, but read through the code and left a question or two. I got tripped up with the compare_column_metadata script a bit, but nothing major I don't think.

[johnclary]

👋 thanks for the feedback. i reworked the logging and added docstrings in socrata_export.py and i refactored _compare_column_metadata.py to be less confusing 👍

edit: and i fixed the bug in the crash list view that had minor injuries taking precedence over serious injuries