search

timgit / pg-boss

Queueing jobs in Postgres from Node.js like a boss
MIT License
2.13k stars 158 forks source link

Understanding `teamSize` and `teamConcurrency` #119

Closed SkyHacks closed 5 years ago

SkyHacks commented 5 years ago

Thank you for this awesome library!

I've spent a lot of time looking at the documentation, issues, and code around the two options: teamSize and teamConcurrency. I'm struggling to see how they impact the functionality of this library.

Can you help me to understand the rationale behind these two options and what are the scenarios where I might consider increasing them from the default values?

At the moment, my use case is mainly pushing scheduled jobs that need to execute after a specific point in time (like dynamically scheduled emails that need to be sent out). I only have one node.js process polling the queue but I would like to be able to safely scale that up at some point to use all of the CPUs on my server without invoking jobs more than once. Is my use case one that would benefit from increasing teamSize and/or teamConcurrency? Does it matter if I'm calling subscribe more than once (to handle different job types)?

timgit commented 5 years ago

You would only use these options if you need to process jobs faster than approx 1 job/second.

SkyHacks commented 5 years ago

One question that I still have is: why limit the number of results returned at all? Why does teamSize default to 1?

timgit commented 5 years ago

pg-boss is pull-based, not push. Every worker node connecting to your database is asking for work to do. teamSize tells pgboss how many jobs you're requesting at once. Each job has an expiration attached to it in order for the supervisor process to know if it has expired. This expiration timer starts once the job has been fetched. If you fetch too many jobs and you're not able to complete them all, some will be marked as failed because of expiration. You can customize all of these settings of course, but I have set the defaults in such a way that you have to opt into increased levels of concurrency. The default expiration is quite long, btw, at 15 minutes, too.

SkyHacks commented 5 years ago

Awesome! Thank you for taking the time to explain that.