search

sfcpc / housing-dashboard

4 stars 0 forks source link

current freshness strategy doesn't work #70

Closed thelee closed 4 years ago

thelee commented 4 years ago

some dates are being populated with speculative future dates and so are not a good signal of the last time a data source has been updated. spend some time thinking about/fixing this.

thelee commented 4 years ago

i think we'll just have to pull in the "last_updated" from schema-less and have to just specify a manual override for the initial table generation. the only reason why "last_updated" doesn't work is because we're working with past data. with regular pipeline runs, last_updated should exactly equal the freshness

thelee commented 4 years ago

no, that doesn't work, because due to latency of some data sources we wouldn't be able to just turn off the override when we launch the actual pipeline