Team Concurrency & Refill in v10

[abdel]

One of the appealing things about PGBoss in v9 for our particular workflow is the ability to configure the job processing concurrency and using the refill option to ensure that there is always a buffer of N jobs ready to go, as jobs may take a different amount of time to process.

We are currently looking to see how we can re-implement that logic following its removal from v10. It seems the only path forward is to go via patch-package to either implement the logic directly in watch or at least expose certain parts of the library (e.g. Manager & Worker) to be able to implement it outside the package.

I was wondering what were the primary reasons for removing the feature, aside from simplifying? Would there be a pathway to restoring the feature or incorporating ways to facilitate consumers implementing it themselves?

[timgit]

Yes, the primary justification in removing that feature was its complexity. Packages like p-map handle the majority of my promise concurrency use cases in my app.

Also, consider that each call to work() adds a new polling worker, so if you need to divide the risk of a single job holding up processing, the multiple worker pattern would at least mitigate a "single point of failure".

[abdel]

Thanks for clarifying! We'll explore a pathway with your suggestion using multiple workers as an alternative, and see how that performs.

[himat]

@timgit

each call to work() adds a new polling worker

If this is true, how are users supposed to have a fixed number of workers instead of having it grow continually? Are we supposed to call work() only N times on our app startup?

[timgit]

Yes