search

andykais / scrape-pages

generalized scraper using a single instruction set for any site that can be statically scraped
https://scrape-pages.js.org
MIT License
6 stars 2 forks source link

Flag metadata.json as in-progress when scraper is running #41

Closed andykais closed 4 years ago

andykais commented 5 years ago

Currently two scrapers can run in parallel on the same folder. This is undesirable since they will both be modifying the same order.

add an "active": true flag to the metadata.json file. It is true once start() has ran, and is false once the done event is emitted. If another scraper tries to run while the active flag is true, then it will error out with the message:

another scraper is running in this location. If you really want to run anyways, add forceStart: true to your params.

The forceStart field works as expected. When it is set, nothing prevents the scraper from running