act-now-coalition / can-scrapers

MIT License
9 stars 13 forks source link

Revert change to NYT scraper that broke in prod. #454

Closed mikelehen closed 1 year ago

mikelehen commented 1 year ago

Reverts part of https://github.com/covid-projections/can-scrapers/pull/452. Not 100% sure how this all works, but it looks like fetch() receives some additional args when running normally which caused stuff to break.

See https://cloud.prefect.io/actnow/task-run/f1aacad5-b2d8-4c90-85ac-f729314af8a2?logs

Flow 'spry-caterpillar-NYTimesCasesDeaths': Task 'fetch': Exception encountered during task execution! Traceback (most recent call last): File "/home/sglyon/miniconda/envs/prefect-can-scrapers/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 884, in get_task_run_state logger=self.logger, File "/home/sglyon/miniconda/envs/prefect-can-scrapers/lib/python3.7/site-packages/prefect/utilities/executors.py", line 468, in run_task_with_timeout return task.run(*args, **kwargs) # type: ignore File "services/prefect/flows/generated_flows.py", line 40, in fetch fn = d._fetch() File "/home/sglyon/can-scrapers/can_tools/scrapers/base.py", line 448, in _fetch data = self.fetch() File "/home/sglyon/can-scrapers/can_tools/scrapers/nytimes/nyt_cases_deaths.py", line 145, in fetch for location_type, files in files.items(): File "/home/sglyon/miniconda/envs/prefect-can-scrapers/lib/python3.7/typing.py", line 706, in getattr return getattr(self.origin, attr) AttributeError: '_SpecialForm' object has no attribute 'items'