City-Bureau / city-scrapers

Scrape, standardize and share public meetings from local government websites
https://cityscrapers.org
MIT License
332 stars 311 forks source link

🕷️ Fix spider: Chi Board of Elections #1057

Closed SimmonsRitchie closed 8 months ago

SimmonsRitchie commented 9 months ago

What's this PR do?

Fixes our Chicago Board of Elections spider (aka. chi_board_elections), which broke due to URL and page structure changes.

Why are we doing this?

We want working scrapers, of course 🤖 The changes in this PR include a rebuilt spider and tests.

Steps to manually test

After installing the project using pipenv (see Readme):

  1. Activate the virtual environment:

    pipenv shell
  2. Run the spider:

    scrapy crawl chi_board_elections -O test_output.csv
  3. Monitor the stdout and ensure that the crawl proceeds without raising any errors. Pay attention to the final status report from scrapy.

  4. Inspect test_output.csv to ensure the data looks valid. I suggest taking a cursory look at the target webpage and clicking through to a few of the meeting detail pages in order to spot check the data.

Are there any smells or added technical debt to note?