search

Police-Data-Accessibility-Project / scrapers

Code relating to scraping public police data.
https://pdap.io
GNU General Public License v3.0
157 stars 33 forks source link

Show status in scrapers table if it fails #142

Closed josh-chamberlain closed 1 year ago

josh-chamberlain commented 2 years ago

Per Stabs' comment below, we should have clone of the Datasets db for Scrapers to use. Scrapers can update the db, which can in turn be synced to DoltHub.

CaptainStabs commented 2 years ago

This one might be difficult to do depending on how we plan on using the scrapers We can't package credentials to our dolthub with the scrapers, so for them to update we will either need to run them from our own server or anyone who runs them will have to either be verified or have to login to dolthub and have it create a PR on it's own (automatically) And while it is possible to automate PR creation (see https://www.dolthub.com/repositories/dolthub/menus/pulls), that in itself isn't perfect as we don't want to spam dolthub with PRs

josh-chamberlain commented 1 year ago

outdated strategy