search

FirebaseExtended / firebase-queue

MIT License
787 stars 108 forks source link

What exactly does "numWorkers" do? #83

Closed holgersindbaek closed 7 years ago

holgersindbaek commented 7 years ago

I have a queue where there's a lot more tasks being added to it than I expected, so I have a long backlog of tasks, that is only getting bigger. That's when I found the "numWorkers" option on firebase-queue. Now I've set it to 2, but I don't fully understand what it does?

I have my queue running on a hobby dyno on Heroku, but from what I know, you can't scale a hobby dyno. If I output "queue.getWorkerCount()", it still shows me 2 though. So what exactly is it that firebase-queue is scaling when I set "numWorkers" to 2 and when will I know when it has reached the limit for what my hobby dyno can take?

holgersindbaek commented 7 years ago

I currently have 800k tasks in my queue and growing, so a bit of clarification would be highly appreciated.

cbraynor commented 7 years ago

First off, apologies for the slow reply

numWorkers is for increasing the number of tasks a single node process will attempt to process asynchronously, and so is good if your tasks are waiting a lot on other processes (network requests etc) but if your process is CPU bound it won't help much.

If you have many CPUs available (I'm not too familiar with Heroku, so I'm not sure) then you might want to consider spawning a child process for each CPU you have available, each one initializing a Queue.