A process which, when run, submits a scraper’s Extraction and metadata to our database.
For now, we're going to use CKAN instead of making our own API from scratch.
Key user story
As a data scraping volunteer, I should be able to run a Scraper from the Scrapers repo and submit the Extraction to PDAP.
Details
We need a place to put Extractions and their Metadata. Once the Extraction is dropped, we should link to its path in the data_intakedatabase.
The simplest, most modern solution is probably an API endpoint.
What's in an Extraction?
The goal: a synchronous bright line between the source material and the scraped result, with the source code thrown in. We can publish these on the website as case studies without fear of legal trouble.
an extraction of "raw files", i.e. no OCR or translation
we don't technically need this as long as we have time stamped version history in github, though that is tougher to untangle and troubleshoot and not as standalone
For now, we're going to use CKAN instead of making our own API from scratch.
Key user story
As a data scraping volunteer, I should be able to run a Scraper from the Scrapers repo and submit the Extraction to PDAP.
Details
We need a place to put Extractions and their Metadata. Once the Extraction is dropped, we should link to its path in the
data_intake
database.The simplest, most modern solution is probably an API endpoint.
What's in an Extraction?
The goal: a synchronous
bright line
between the source material and the scraped result, with the source code thrown in. We can publish these on the website ascase studies
without fear of legal trouble.scraper.py
code itself (nice to have)Visual aid
https://pdap.invisionapp.com/freehand/Data-intake-flow-Q01qjpCvN
To do