Ecocounter capture backlogged data

CityofToronto / bdit_data-sources

Data sources used by the Big Data Innovation Team

https://github.com/orgs/CityofToronto/teams/bigdatainnovationteam

GNU General Public License v3.0

38 stars 8 forks source link

Ecocounter capture backlogged data #984

Closed gabrielwol closed 2 months ago

gabrielwol commented 3 months ago

According to Ecocounter, data can sometimes arrive late due to poor cellular connection. We should add a process to retry sensors missing data on previous days (up to a week?).

gabrielwol commented 3 months ago

However I'm assuming changing the ecocounter_pull time to 10am as part of #983 will solve most of the issue so we should evaluate if it's still necessary after that change.

chmnata commented 3 months ago

Would it make sense to have all locations as separate tasks with retries?

gabrielwol commented 3 months ago

Definitely open to that!

gabrielwol commented 3 months ago

Here's an example of a sensor that reported 5 days of backlogged data at once.

gabrielwol commented 3 months ago

I successfully tried out dynamically mapped tasks, however need to think more about:

how to implement slack alerts (consolidated alert task like in replicator)
how many retries
add info about which sites are deprecated to not pull those

gabrielwol commented 2 months ago

Chris identified another example which can be used for testing; site_id = 300028433 (Yonge St, South of Davisville Ave) transmitted recent data in batches (07/08 and 07/14): image (6)