search

openva / rs-machine

Richmond Sunlight's scrapers, parsers, etc.
MIT License
0 stars 1 forks source link

Incorporate summaries bulk data #39

Open waldoj opened 6 years ago

waldoj commented 6 years ago

Right now we're scraping every summary. The legislature has started publishing Summaries.csv on their FTP site, with these headers:

"SUM_BILNO","SUMMARY_DOCID","SUMMARY_TYPE","SUMMARY_TEXT"

Get all summaries from this file, and stop scraping them with summaries.php.

waldoj commented 6 years ago

Hang on—I'm dubious that the hash-checking system is working here.