Remove CDCCovidDataTracker, add caching for large scrapers, increase time between MainFlows

act-now-coalition / can-scrapers

MIT License

9 stars 13 forks source link

Closed smcclure17 closed 2 years ago

smcclure17 commented 2 years ago

removes the CDCCovidDataTracker and the Prefect flows related to it
increases MainFlow schedule from 3 hours to 4 hours
adds etag checking to large scrapers to only pull in data if it has been updated
adds a timeout to the put() task to fail scrapers if they get stuck