search

CoalitionBuilder / okcongress

Find the co-sponsor for your next bill
http://www.coalition-builder.com/
GNU General Public License v3.0
7 stars 0 forks source link

Move the CSV of Legislators into a daily update #10

Open e01010100 opened 9 years ago

e01010100 commented 9 years ago

Currently the CSV file of legislator information has been converted to JSON and committed to our repository. This needs to be moved so that it is pulled from OpenGov on a regular basis and cached. Additionally, there needs to be a way to dump & reset the cache but that doesn't have to be implemented now as someone could DOS the site by constantly hitting that endpoint.

benjaminxscott commented 9 years ago

Ah so basically pull it into the running instance rather than statically including in the repo

Does opengov respond with a csv or json? edit: csv

Should probably do integrity / sanity check of received file

Suggest implementing the 'reload functionality' as a grunt task, so it can be run locally but not remotely (nice for debugging, no remote DoS)

e01010100 commented 9 years ago

yeah, more specifically we want to pull it into a daily cache. The original source was/is a CSV file saved on github The list of committees is a Yaml file saved on github except it was a broken yaml file that I had to manually fix.

In both cases we should do at least some integrity checking but there isn't a whole lot we can do that I can think of off the top of my head.

benjaminxscott commented 9 years ago

yeah i'd imagine a grunt task using this lib to rebuild the cache every day, and run some checks on random committees / legislators