Police-Data-Accessibility-Project / scrapers

Code relating to scraping public police data.
https://pdap.io
GNU General Public License v3.0
157 stars 35 forks source link

Bulk Download of CityProtect #50

Closed EricTurner3 closed 3 years ago

EricTurner3 commented 3 years ago

This script will download ALL available data from CityProtect's public endpoint for ALL agencies that have bulkDownload enabled. It will take several hours to run, about 355 agencies have that flag, some with 10+ csvs each. Attached are some samples from a variety of states.

As the endpoints are open with no auth, and bulk download is allowed, this should be authorized.

I put a sleep(5) in between files to prevent a DoS attack from downloading too many files at once.

Note: Some data is also from Canadian provinces.