search

City-Bureau / city-scrapers-omaha

City Scrapers for Omaha
MIT License
0 stars 0 forks source link

🕷️ Fix spider: Omaha Planning Department: Steamfitters/Pipefitters Board #70

Closed SimmonsRitchie closed 1 month ago

SimmonsRitchie commented 1 month ago

What's this PR do?

Fixes our Omaha Planning Department: Steamfitters/Pipefitters Board spider (aka. oma_planning_exam_pipefitters).

Why are we doing this?

The spider broke due to changes on the pages it's targeting. The changes in this PR ensure the scraper runs without error.

Steps to manually test

After installing the project using pipenv:

  1. Activate the virtual environment:

    pipenv shell

  2. Run the spider:

    scrapy crawl oma_planning_exam_pipefitters -O test_output.csv

  3. Monitor the stdout and ensure that the crawl proceeds without raising any errors. Pay attention to the final status report from scrapy.

  4. Inspect test_output.csv to ensure the data looks valid. I suggest opening a few of the URLs under the source column of test_output.csv and comparing the data for the row with what you see on the page.

Are there any smells or added technical debt to note?