search

FirebaseExtended / firebase-queue

MIT License
786 stars 108 forks source link

Timedout Queues #85

Open ahaverty opened 7 years ago

ahaverty commented 7 years ago

Setting numWorkers to 100, and running tasks until they timeout, appears to be not releasing queues. I haven't isolated code to prove yet, but I wanted to ask for more detail about how timeouts are intended to be used.

Is the above by design, should I be manually timing out my tasks and resolving like the guide example does, or is this a bug?

ahaverty commented 7 years ago

For example, the first 100 tasks appear to be taken and timedout, but nothing after that first 100 is picked up by a queue

cbraynor commented 7 years ago

100 workers is a large number of tasks to be processed in parallel in a single node process for most tasks (unless they're highly async). First off, I'd suggest lowering the number of workers per process, and try running multiple node processes, or spawning child processes to do the processing.

Let me know if you still have issues after trying that - you might simply be trying to do too much on a single CPU

cbraynor commented 7 years ago

the 1.6.1 release fixed a bug in timeouts that could be related to some of these symptoms. Can you check out the new release and report back if things still don't work as you expect