Closed will-snavely closed 3 years ago
Note: I propose allowing a dev to stand up their own config find in the city_scrapers
folder, called dev.py
(opposed to the current prod.py
). That's what I did for the purposes of testing, here. This is why this commit includes a .gitignore
for that file.
This looks good. Thanks @will-snavely !
The
BethelParkSpider
was failing in production for a few reasonscity-scrapers-core
code; it compares the start time of the meeting withdatetime.now()
, a tz-unaware date. If the start time is tz-aware, this comparison fails. I'm not sure their code is correct, though).None
I made these changes and ensured the scraper now produces output in a production setting (by running
scrapy crawl
and dumping to my own s3 bucket). Also updated tests to check these new requirements.