search

opengovernment / askthem

Monitor and interact with local governments in the United States
GNU Affero General Public License v3.0
41 stars 15 forks source link

Incorporate open civic data municipal scrapers? #17

Closed derekeder closed 6 years ago

derekeder commented 11 years ago

These guys seem to be pushing on some of the same wheels we are. Definitely worth checking out: https://github.com/opencivicdata/municipal-scrapers-us

davidmooreppf commented 11 years ago

Yep of course that's the Sunlight Open States peeps, w/ the support from Google.org to bring down to municipal - that would work for me, as another source for city councilmembers. We can check in w/ James & Paul directly of course. Good find, I hadn't seen their dozen or so cities yet.

walter commented 10 years ago

I took a look at the scrapers. They rely on the python version of pupa. After playing with the python version, my preference is for OpenNorth's fork of pupa in ruby.

The scrapers are city by city and each city's coverage varies (some have events/meetings and councilors, others have only one aspect). It's probably not going to be a "bulk" solution to our municipal data needs as it stands.

However they could be a building block for a toolkit that we distribute to municipal opendata developers to help us extend our dataset.

In short, worth keeping an eye and possibly useful in the long run.

walter commented 10 years ago

Currently planning to replace scraped_local_gov use with opencivicdata municipal scrapers.

Decided to stick with the python version of pupa since all the US municipal scrapers are done in python.

walter commented 10 years ago

Will be used to do the work for opengovernment/product/issues/19.