andykais / scrape-pages

generalized scraper using a single instruction set for any site that can be statically scraped
https://scrape-pages.js.org
MIT License
6 stars 2 forks source link

make emitter the authority on any scraper commands/messages #46

Open andykais opened 4 years ago

andykais commented 4 years ago

The idea behind this is to allow a single channel for communication between the scraper internals and the library's user. The event emitter. This is cool for a number of reasons, but the biggest is that it makes communication across workers extremely easy. Masters and workers can both instantiate a scraper but the master just passes on commands to the worker. It would look a bit like this

// worker
const scraper = new ScraperProgram(config, options, params)
process.on('message', scraper.emit)
scraper.onAny(process.send)

// master
const worker = new Worker()
const scraper = new ScraperProgram(config, options, params)
const controller = new ScraperController(scraper)
worker.on('message', this.sendToScraper)
this.onMsgFromScraper(worker.send)
await controller.start()
await controller.stop()
controller.on('done', () => console.log('Done.'))

We can keep all the classyness that makes accessing the library nice but the class methods will just call emitter endpoints.