City-Bureau / city-scrapers-cle

City Scrapers project for Cleveland
https://cityscrapers.org/
MIT License
15 stars 14 forks source link

🕷️ Fix spider: cuya_board_revision #80

Closed SimmonsRitchie closed 10 months ago

SimmonsRitchie commented 10 months ago

What's this PR do?

Fixes our Cuyahoga County Board of Revision spider (aka. cuya_board_revision), which broke due to URL and page structure changes across Cuyahoga County's website.

Why are we doing this?

We want working scrapers, of course 🤖 The changes in this PR include use of a new class mixin for scraping the Cuyahoga County website.

Steps to manually test

After installing the project using pipenv (see Readme):

  1. Activate the virtual environment:

    pipenv shell
  2. Run the spider:

    scrapy crawl cuya_board_revision -O test_output.csv
  3. Monitor the stdout and ensure that the crawl proceeds without raising any errors. Pay attention to the final status report from scrapy.

  4. Inspect test_output.csv to ensure the data looks valid. I suggest opening a few of the URLs under the source column of test_output.csv and comparing the data for the row with what you see on the page.

Are there any smells or added technical debt to note?