The CDC county-level data endpoint doesn't send an Etag header (which we use for all our other caching), but we can use the runid field in the response. To do so, we need to lift/restructure some of our existing logic
Testing:
To test the caching, run the following twice:
from can_tools.scrapers import CDCCountyCasesDeaths
CDCCountyCasesDeaths().check_if_new_data_and_update() # should return True the first time, False thereafter
We can spin up a local Prefect instance and test the workflow end-to-end. I'd be happy to document these steps if we fill that would be advantageous
The CDC county-level data endpoint doesn't send an
Etagheader (which we use for all our other caching), but we can use therunidfield in the response. To do so, we need to lift/restructure some of our existing logicTesting:
We can spin up a local Prefect instance and test the workflow end-to-end. I'd be happy to document these steps if we fill that would be advantageous