Numerous database queries

[gavacho]

Hi, many thanks for pg-boss.

We're having an issue with the number of database queries being made by pg-boss. Our app is subscribing to 13 pg-boss queues. When we scale up to 7 instances of our app, our database starts getting about 100 queries a second.

We had been using the default value for newJobCheckInterval but we'll adjust that in order to decrease the number of queries per second. But this solution has the unintended consequence of running jobs less frequently.

Would it be possible to have 1 query made every newJobCheckInterval instead of a query for every worker/subscription?

[timgit]

Hi, you're welcome!

There are a lot of options available in the configuration to support a variety of use cases. Following are some to consider, but this is not exhaustive, as you may be able to dream up more.

1. Name all queues with a similar naming convention and subscribe to a wildcard. Once the job is retrieved, use the actual job name and call the appropriate handler.
2. Use the teamSize or batchSize option to fetch jobs in a batch per interval. Keep expiration in mind when increasing this as all jobs in the batch need to be completed within your expiration configuration.
3. Check the queue size during a subscription and re-subscribe with a customized fetch interval to dynamically adjust to spikes in job volume.
4. Use fetch() and complete() to bypass subscribe() with custom logic with a combination of any of the above options.

I'll add that we regularly subscribe to hundreds (and sometimes thousands) of queues across multiple instances without a significant drop in performance. If you're experiencing issues at this volume, please share your performance metrics.

[Israel001]

@timgit Apart from the first option, the others sounds complicated. An example would be nice (:

[timgit]

There are a couple in the examples file