search

moj-analytical-services / airflow-matrix-scraper

scraper for matrixbooking api
0 stars 2 forks source link

Promote historic dev locations data to prod #56

Closed gwionap closed 4 months ago

gwionap commented 4 months ago
williamorrie commented 4 months ago

This relates to #54

matrix_dev.locations has an earliest consistent location count (approx. 4900) from an ingestion date of 2024-03-25, location data from earlier ingestions have a much smaller count, suggesting that the extraction code prior to this was incorrect. matrix_prod.locations has an earliest consistent location count (approx. 5215) from an ingestion date of 2024-05-01

Laurence Droy has asked that prod data be updated with the dev data as there is an extra month of historical data

williamorrie commented 4 months ago

Manually updated the files from console earlier today, feeling bad now 😞, so have scripted the update for future ref

PR - https://github.com/moj-analytical-services/airflow-matrix-scraper/pull/57