Open Data Network Scraper

Police-Data-Accessibility-Project / scrapers

Code relating to scraping public police data.

https://pdap.io

GNU General Public License v3.0

157 stars 35 forks source link

Open Data Network Scraper #228

Closed EvilDrPurple closed 1 year ago

EvilDrPurple commented 1 year ago

Fixes

Description

Script retrieves relevant data from Open Data Network and writes it to a CSV to be uploaded to our data sources DB
One part currently checks for duplicate entries from our DB using a local CSV of our current DB (if the file exists). This could be implemented with the API instead once that is set up so that we don't accidentally upload duplicate records.
As I mentioned in the comments of #204, some of the data from ODN is inconsistent or left out entirely.

EvilDrPurple commented 1 year ago

This works for me, but it needs a little readme.

My steps were:
1. clone the repository, or download the file

2. `cd` into `open_data_network`, or the repository containing the scraper file

3. run `python3 open_data_network`
It should probably discuss any system requirements (python installed).

@josh-chamberlain Alright, how does the README look? Anything else you want changed?

josh-chamberlain commented 1 year ago

looks good, passing to @mbodeantor

Police-Data-Accessibility-Project / scrapers

Open Data Network Scraper #228

Fixes

204

Description