Fixes our Cuyahoga County Board of Health spider (aka. cuya_health), which has been raising errors in Sentry for some time.
Why are we doing this?
We want working scrapers, of course 🤖 The changes in this PR include modifications to ensure the scraper functions without error. Specifically, this PR the PDF parsing and spider idle logic and refactors to simply parse the URLs on the agenda/meetings page and then group them together.
Steps to manually test
After installing the project using pipenv:
Activate the virtual environment:
pipenv shell
Run the spider:
scrapy crawl cuya_health -O test_output.csv
Monitor the stdout and ensure that the crawl proceeds without raising any errors. Pay attention to the final status report from scrapy.
Inspect test_output.csv to ensure the data looks valid. I suggest opening a few of the URLs under the source column of test_output.csv and comparing the data for the row with what you see on the page.
Are there any smells or added technical debt to note?
At present, there aren't any 2024 meetings for this agency. We'll assume those will appear eventually.
What's this PR do?
Fixes our Cuyahoga County Board of Health spider (aka.
cuya_health), which has been raising errors in Sentry for some time.Why are we doing this?
We want working scrapers, of course 🤖 The changes in this PR include modifications to ensure the scraper functions without error. Specifically, this PR the PDF parsing and spider idle logic and refactors to simply parse the URLs on the agenda/meetings page and then group them together.
Steps to manually test
After installing the project using
pipenv:Activate the virtual environment:
Run the spider:
Monitor the stdout and ensure that the crawl proceeds without raising any errors. Pay attention to the final status report from scrapy.
Inspect
test_output.csvto ensure the data looks valid. I suggest opening a few of the URLs under the source column of test_output.csv and comparing the data for the row with what you see on the page.Are there any smells or added technical debt to note?