search

thomasdondorf / puppeteer-cluster

Puppeteer Pool, run a cluster of instances in parallel
MIT License
3.24k stars 310 forks source link

Better queue management #272

Open vickyRathee opened 4 years ago

vickyRathee commented 4 years ago

I see you are working on new query the status API. Can we use that to get how many jobs in queue and cancel the job?

  1. I am using express API and don't want to queue the job if there are more then 10 (n) jobs already in queue. So better to throw error, instead gateway timeout.
  2. In another case, if I queued a job and it didn't completed in 60 (n) seconds. Can I cancel it using this new API? I see the job is still running on backend while http client timed out. Because HTTP client is disconnected with express timeout (502 - gateway timeout) error after 30-60 seconds and it's doesn't make sense to complete that job until I am using a database to store the result of job.
  3. Can I get the jobId when running cluster.execute() or cluster.queue() to store the result in database when job is completed.
thomasdondorf commented 4 years ago

1+2 -> Yes, the idea is to support these use cases.

3 -> This can already be done right now. Just pass the extra information (job id) as data to the job.

vickyRathee commented 4 years ago

Thanks, I got the #3 working. Waiting for better queue management API.