Calls for service scraper experiment - Githubissues

Police-Data-Accessibility-Project / scrapers

Code relating to scraping public police data.

https://pdap.io

GNU General Public License v3.0

157 stars 35 forks source link

Calls for service scraper experiment #189

Closed josh-chamberlain closed 1 year ago

josh-chamberlain commented 1 year ago

Latest

We're likely going to do an experiment with GitHub Actions scraping into a GitHub repo. I'm still waiting to check in with the original data requestors to see how they plan to use it.

Open questions:

[ ] where does the data go?
[ ] who does maintenance?
[ ] how big is the data?

Background:

https://discord.com/channels/828274060034965575/1034159909635358782

What to scrape:

airtable data source
http://gisapps1.mapoakland.com/callsforservice/
- scrapable with esri2geojson: http://gismaps.oaklandca.gov/oaklandgis/rest/services/callforservice_2015_FC/FeatureServer/0
- can be added to qgis and exported as CSV
- example from one day: cfs_1.csv

Scraper hosting options

github actions
digitalocean
aws lambda / ec2
jacob to present guidance on this

Data storage options

github repo (scraping to git)
https://www.wrgl.co/
https://www.documentcloud.org/
emailed to someone / a list of people daily

josh-chamberlain commented 1 year ago

closed by github-actions-demo/csv at main · Police-Data-Accessibility-Project/github-actions-demo