CivicTechTO / github-scraper

A scraper written for my reseach project on civic hacking.
2 stars 1 forks source link

Run scraper regularly #4

Open patcon opened 5 years ago

patcon commented 5 years ago

Thinking I could get this running nightly, and then start storing open datasets in Google Sheets. This allows them to be pretty easily imported into a neo4j sandbox for exploration and linking within the community.

Could probably use either travisci, or using heroku scheduled tasks, not unlike how OpenNorth's Represent API does it: https://github.com/opennorth/scrapers_ca_app https://scrapers.herokuapp.com/

Manually uploaded dataset: https://docs.google.com/spreadsheets/d/1TuFhZbRihYx-WgKk1YD4a-0m-rrTq0O69PJ-4p7xAqA/edit#gid=213514514

Demo of using GSheet sandbox: https://www.youtube.com/watch?v=7aON114bXxA