Is it possible to use this library to create a pool of long running phantomjs processes that runs tasks off a queue of sites to scrape?
I would like to have a pool of 10 processes, and use potentially a redis queue to queue up tasks. Then one by one the tasks would be finished by the phantomjs processes and they will then return the scrape to either another queue or send it back to the initiating process.
Is it possible to use this library to create a pool of long running phantomjs processes that runs tasks off a queue of sites to scrape?
I would like to have a pool of 10 processes, and use potentially a redis queue to queue up tasks. Then one by one the tasks would be finished by the phantomjs processes and they will then return the scrape to either another queue or send it back to the initiating process.