pgh-public-meetings / city-scrapers-pitt

Pittsburgh City Scrapers: sourcing public meetings in Pittsburgh
https://pgh-public-meetings.github.io/events/
MIT License
19 stars 66 forks source link

Fixing issues with BethelParkSpider (prod failures) #179

Closed will-snavely closed 3 years ago

will-snavely commented 3 years ago

The BethelParkSpider was failing in production for a few reasons

I made these changes and ensured the scraper now produces output in a production setting (by running scrapy crawl and dumping to my own s3 bucket). Also updated tests to check these new requirements.

will-snavely commented 3 years ago

Note: I propose allowing a dev to stand up their own config find in the city_scrapers folder, called dev.py (opposed to the current prod.py). That's what I did for the purposes of testing, here. This is why this commit includes a .gitignore for that file.

ben-nathanson commented 3 years ago

This looks good. Thanks @will-snavely !