search

Police-Data-Accessibility-Project / scrapers

Code relating to scraping public police data.
https://pdap.io
GNU General Public License v3.0
157 stars 33 forks source link

Wire up Scraping app CI to the Scrapers table #127

Closed josh-chamberlain closed 1 year ago

josh-chamberlain commented 3 years ago

https://www.dolthub.com/repositories/pdap/datasets/data/master/scrapers

As a scraper volunteer, I need to update this scrapers table using a variety of local utilities.

It should be updated by the utility @CaptainStabs made, or the intake utilities, or similar.

For more context, discussion: https://discord.com/channels/828274060034965575/828283062827221083/860953464527388713

CaptainStabs commented 3 years ago

Programmatically creating PRs on dolt is all done through GraphQL queries. I'm not entirely sure how we would provide an easy way for volunteers to feed the creation utility their cookie. The other issue with creating a scraper ID in dolt via the utility is that it's not accessible for someone who just wants to create a single scraper (have to download more software). This may be more of a task for us to do automatically behind the scenes (git diff to find the new folder(s))?, generate a guuid, perhaps add author(s) to the scraper table, and then insert it.

josh-chamberlain commented 1 year ago

outdated, not storing stuff in Dolt