City-Bureau / city-scrapers

Scrape, standardize and share public meetings from local government websites
https://cityscrapers.org
MIT License
332 stars 310 forks source link

0808-spider-il_elections #938

Closed burakcank closed 4 years ago

burakcank commented 4 years ago

Summary

Issue: #808

Checklist

Questions

PR for counseling.

pjsier commented 4 years ago

Is there anything specific you'd like me to look at? In general we want the PR template checklist to stay in the description even if it's not all checked yet, and for the CI checks to pass

burakcank commented 4 years ago

PR template added to the description, CI checks pass for now.

Help needed on handling multiple URLs :)

pjsier commented 4 years ago

For parsing multiple URLs, you're going to want to use the page of meeting minutes as your start URL. Then in parse you can parse the meeting minutes into a dictionary that stores each date as a key and the link as a value. From this parse method you can yield a request to the agenda page, and while you're parsing each meeting you can pull the minutes link from that dictionary if it's present. Here's an example https://github.com/City-Bureau/city-scrapers/blob/00bc430771c2a7fc5fee456bdde83bbf65c41164/city_scrapers/spiders/chi_teacherpension.py

burakcank commented 4 years ago

@pjsier Waiting for feedback if I missed some minor details or something, other than that it looks completed I reckon.

burakcank commented 4 years ago

Done :+1: