City-Bureau / city-scrapers

Scrape, standardize and share public meetings from local government websites
https://cityscrapers.org
MIT License
329 stars 310 forks source link

🕷️ Fix spider: Illinois Commerce Commission #1054

Closed SimmonsRitchie closed 6 months ago

SimmonsRitchie commented 7 months ago

What's this PR do?

Fixes our Illinois Commerce Commission spider (aka. il_commerce), which broke due to URL and HTML changes on the target webpage.

Why are we doing this?

We want working scrapers, of course 🤖 The changes in this PR include targeting a slightly different URL and using new CSS selectors.

Steps to manually test

After installing the project using pipenv (see Readme):

  1. Run:

    scrapy crawl il_commerce -O test_output.csv
  2. Monitor the stdout and ensure that the crawl proceeds without raising any errors. Pay attention to the final status report from scrapy.

  3. Inspect test_output.csv to ensure the data looks valid. I suggest taking a cursory look at the target webpage and clicking through to a few of the meeting detail pages in order to spot check the data.

Are there any smells or added technical debt to note?

LienDang commented 6 months ago

Looks good to me!

SimmonsRitchie commented 6 months ago

Thanks for the review, @LienDang! I appreciate it.