search

timgit / pg-boss

Queueing jobs in Postgres from Node.js like a boss
MIT License
2.04k stars 157 forks source link

Question: Multiple worker servers processing single queue name #218

Closed Ollie1700 closed 3 years ago

Ollie1700 commented 3 years ago

Hi guys 👋 Sorry for opening an issue for this but hopefully it's an easily answered question.

We have a system in place where business calculations are published to a single queue and we're at the point where sometimes backlog can be tens of thousands of jobs.

Business calculations take ~2 seconds so to avoid hours and days of backlog we're now looking into spinning up more worker instances based on number of jobs in the backlog.

So my question is, when spinning up more than one server that is subscribed to the same queue name, what important considerations are to be made in regards to pg-boss? Does pg-boss guarantee that a job will only be run once, or should we plan to account for this manually?

Apologies if this is a really simple one - I couldn't uncover too much detail regarding multiple worker systems in the documentation (hopefully I didn't just completely miss it!)

Thank you in advance!

timgit commented 3 years ago

Yes, you can create as many workers on the same queue as needed.

Ollie1700 commented 3 years ago

Thank you for the confirmation!