search

cityofaustin / atd-data-tech

Austin Transportation Data & Technology Services
17 stars 2 forks source link

[Enhancement] Break VZ Socrata ETL DAG dataset replacements into separate tasks #13139

Open mddilley opened 11 months ago

mddilley commented 11 months ago

The VZ Socrata ETL DAG updates both the crash and people datasets all in one task. It could be beneficial to break these crash and people dataset replacements/updates into two separate tasks. The crash dataset takes much longer to run. If an issue occurs during the people dataset update, the whole DAG must restart and could delay patching the datasets to get Vision Zero Viewer back online (since it is powered by this dataset).

See for new info about re-running individual tasks: https://austininnovation.slack.com/archives/CHZE6BC6L/p1690903981451939

johnclary commented 4 weeks ago

Tagging with layer data model because i think we'll almost definitely take this on as part of that work!