Feature request: singletonQueued -> singletonNextSlot throttled by job status created and retry

[tcoats]

Can we introduce a new parameter, something like singletonQueued bool? This defaults false, but when true allows an alternate way to throttle other than time. When true the job status is used for singleton detection - only one singletonKey can exist within jobs of statuses created or retry. It could be used with singleton time parameters too, further narrowing the window where singletons are enforced.

We want to throttle (singleton), but once a job has started executing there should be another job queued because the database read at the start of the worker may now be out of date, especially if the job is long running.

Thoughts? Happy to implement this if it sounds suitable. It will make the createJob sql and logic more complex. More running in Postgres.

[timgit]

This sounds like sendSingleton(). Did you already try that?

[tcoats]

It's close. When a job is active we'd like to be able to add another job to created. The job_singletonKey index stops that I think. It would be great if it was state < 'active' instead of state < 'completed' - I think, not 100% about these status range comparisons.

Unless I'm missing something? We've been writing some test code that doesn't send jobs when singletonKey is set the same but one of the jobs is 'active'.

[timgit]

sendSingleton() uses a magic string to opt into a different unique constraint for < 'active'

[BNemhauser]

Hello Tim

I am one of Thomas's employees. Thank you for pointing out sendSingleton(), it gets quite close to what we want, however (without going into too much detail) losing control over the singletonKey is a bit of a deal breaker for us.

Would it be possible to get access to this jobState < 'active' behavior with a different argument to the send function?

Cheers, Bryce

[timgit]

When I implemented sendSingleton(), it was with a concern that perhaps my original implementation of throttling was not as useful as I had originally planned. :crying_cat_face: It seems to most valuable approach is to consolidate the focus of the system to < active and make < completed obsolete.

[BNemhauser]

Good morning Tim, That would indeed be quite helpful for us!

Maybe it is worth considering making the guarded state configurable by parameter to the send function? While it does add a little bit more piping, it would allow people to continue to guard against the 'completed' state if they wanted to and saving your existing implementation from being made obsolete. Just an idea, not sure how well it would fit with your vision for the project.

Cheers, Bryce

[jasir]

Hello Tim, thanks for really great lib! I have similar needs (to be able to add job if job is already in active state). If I have understood well, you are suggesting this change? It works for my needs pretty good.

CREATE UNIQUE INDEX job_singletonkey
  ON syncboss.job(name, singletonkey)
  WHERE ((state < 'active'::syncboss.JOB_STATE) AND (singletonon IS NULL) AND
         (NOT (singletonkey = '__pgboss__singleton_queue'::TEXT)));

[timgit]

The queue policy stately added in v10 provides this.