The end goal: a magic button for any Scraper that says something like Run Scraper Locally. When this button is clicked, the user needs to do as little as possible for the scraper to run and give them an Extraction. Allows a user to both donate compute time to PDAP and run scrapers for their own benefit.
If we have a Scraper written for a Data Source, and we've created an Archive of the Data Source, we should allow people to run that Scraper locally on demand. They will use their own compute power.
Can we write a package or plugin that lets anyone run our scrapers in-browser?
This is achieved adding things to the existing PDAP-app repo and probably deploying it to app.pdap.io or a local version.
This may be some kind of Docker file.
The package should include all necessary dependencies.
It could include a local version of data sources search
Users should be able to "Run Scraper Locally" on any Dataset they find that has a Scraper.
The end goal: a magic button for any Scraper that says something like
Run Scraper Locally. When this button is clicked, the user needs to do as little as possible for the scraper to run and give them an Extraction. Allows a user to both donate compute time to PDAP and run scrapers for their own benefit.If we have a Scraper written for a Data Source, and we've created an Archive of the Data Source, we should allow people to run that Scraper locally on demand. They will use their own compute power.
Can we write a package or plugin that lets anyone run our scrapers in-browser?
This is achieved adding things to the existing PDAP-app repo and probably deploying it to app.pdap.io or a local version.
This may be some kind of Docker file.
The package should include all necessary dependencies.
It could include a local version of data sources search
Users should be able to "Run Scraper Locally" on any Dataset they find that has a Scraper.
The Extractions should be saved locally.