City Scrapers St. Louis
What are the City Scrapers and why do we want them?
Public meetings are important spaces for democracy where any resident can participate and hold public figures accountable. But how does the public know when meetings are happening? It isn’t easy! These events are spread across dozens of websites, rarely in useful data formats.
Our Mission
The mission of the City Scrapers project is to increase access and transparency around public meetings across the St. Louis County by making it easier for everyone to know when and where public meetings are held.
All of the meetings gathered by our spiders can be viewed here!
What can I learn from working on the City Scrapers?
A lot about the City of St. Louis (and other municipalities of the Greater St. Louis area)! What is City Council talking about this week? What are the local school councils, and what community power do they have? What neighborhoods is the police department doing outreach in? Who governs our water?
From building a scraper, you'll gain experience with:
- How the web works (HTTP requests and responses, reading HTML)
- Writing functions and tests in Python
- Version control and collaborative coding (git and Github)
- A basic data file format (JSON), working with a schema and data validation
- Problem-solving, finding patterns, designing robust code
Contributing
We welcome both coders and non-coders to help out with our project!
- Fill out this form to join our Slack channel and meet the community!
- Read about how we collaborate for more details.
Don't see your local public meetings?
Fill out this form to join our Slack channel! We love hearing from the community and learning about how we can better serve our city.
If there are any public meetings that you would like us to create a scraper for, please fill out this form to make a request.
When reviewing scraper requests, we might consider things such as:
- Are these one-off meetings or recurring?
- If they are one-off meetings, do we expect more in the future to be announced using a similar structure?
- Is there historical data that could also be scraped using the same spider and might that be useful?
- What is the estimated time and effort to write the scraper vs manual entry (i.e. if it takes 2-3 minutes to manually enter a single meeting and there are x number of meetings, how does that compare to the time taken to write the scraper)
Notes
This project is based off of a template repo provided by City Bureau. You can read more about what they do at citybureau.org.
We would also like to thank Pat at City Bureau for his patience and help setting up this project.