Use incremental loading to publish crash data - Githubissues

cityofaustin / atd-data-tech

Austin Transportation Data & Technology Services

18 stars 2 forks source link

Use incremental loading to publish crash data #20042

Open johnclary opened 20 hours ago

johnclary commented 20 hours ago

We currently do a full truncate/replace of our public crash datasets on a nightly basis. This is error-prone and can leave the VZV in an error state.

We should switch to incremental publishing of crash records, in which only modified records are published. To do this, we need to:

Update our crash db views to have a modified date column that accounts when the crash record or any of its related records were modified
Update the socrata db views to include soft-deleted records
Update the VZV to exclude soft deleted records