search

Police-Data-Accessibility-Project / scrapers

Code relating to scraping public police data.
https://pdap.io
GNU General Public License v3.0
157 stars 33 forks source link

Make a place (Hadoop) to store non-csv data #135

Closed josh-chamberlain closed 2 years ago

josh-chamberlain commented 2 years ago

We need to keep our own audit trail. We’d probably just want the ETL library to drop the file on the file system and we store a record in the database about the scraper it came from and when (and who?)

josh-chamberlain commented 2 years ago

@richardji7 I think this one can be closed, anything we should be sure to document in the Notion?

josh-chamberlain commented 2 years ago

https://docs.pdap.io/components/data-storage