search

City-Bureau / city-scrapers

Scrape, standardize and share public meetings from local government websites
https://cityscrapers.org
MIT License
329 stars 311 forks source link

🕷️ Fix spider: Chicago Low-Income Housing Trust Fund #1122

Closed SimmonsRitchie closed 2 months ago

SimmonsRitchie commented 2 months ago

What's this PR do?

Fixes our Chicago Low-Income Housing Trust Fund spider (aka. chi_low_income_housing_trust_fund), which broke due to page structure changes.

Why are we doing this?

We want working scrapers, of course 🤖 The spider now targets an icalendar file with meeting data.

Steps to manually test

After installing the project using pipenv:

  1. Activate the virtual environment:

    pipenv shell

  2. Run the spider:

    scrapy crawl chi_low_income_housing_trust_fund -O test_output.csv

  3. Monitor the stdout and ensure that the crawl proceeds without raising any errors. Pay attention to the final status report from scrapy.

  4. Inspect test_output.csv to ensure the data looks valid. I suggest opening a few of the URLs under the source column of test_output.csv and comparing the data for the row with what you see on the page.

Are there any smells or added technical debt to note?