A complete data set of all US municipal gov't web site URLs

[daguar]

BLUF: This is surprisingly hard to come by — some resources exist, but nothing comprehensive as far as I can tell. Having this as a data set folks could contribute to, and maybe an API, would be neat.

(MOSTLY A STUB — capturing some work/research I did in my spare time this weekend before I close those tabs)

• Core data set would be the Census of Governments, which has unique IDs and crosswalks (FIPS) for the ~90k governmental entities in the US
  • The 2007 version of this has a complete data set available in the "Governments Integrated Directory" (GID) http://harvester.census.gov/gid/gid_07/options.html — unclear if this exists for the 2012 census
  • Note: the GID above contains URLs! It's just from 2007, so lots are missing and potentially changed since.
  • To pull down the data as XLS, best approach is to select a category and select "all states" and it'll just give you a zip file

Pretty easy to pull down search engine results using a little script fed by the GID data set; could also have lots of folks contribute.

Related Open Data StackExchange thread: http://opendata.stackexchange.com/questions/7/u-s-city-and-county-web-data-api/168#168

• Potentially related projects:
  • "DemocracyMap" (from @philipashlock ): http://pages.e-democracy.org/DemocracyMap (unclear if this has URLs and what its status is)
  • Sunlight's Open Civic Data Project ( @opencivicdata @jamesturk @paultag ): http://opencivicdata.org/developers/ (I don't think this has URLs)
  • Google's Civic Information API: https://developers.google.com/civic-information/ (pretty sure this doesn't have it, but they might be interested or have it internally somewhere)

[daguar]

SBA also has an API with some of this data, but the StackExchange thread above implies it's pretty incomplete: https://www.sba.gov/about-sba/sba_performance/sba_data_store/web_service_api/u_s_city_and_county_web_data_api

[daguar]

Also a data source http://www.statelocalgov.net/local-ca.cfm

[asweigart]

Are you still working on this?

Here's a list of all .gov domain names: https://18f.gsa.gov/2014/12/18/a-complete-list-of-gov-domains/