riverqueue / river

Fast and reliable background jobs in Go
https://riverqueue.com
Mozilla Public License 2.0
3.59k stars 94 forks source link

[Feature Request] Job throttling #672

Open profsmallpine opened 2 weeks ago

profsmallpine commented 2 weeks ago

This might already be possible, but I haven't been able to figure it out after reading the docs. If I only want a specific type of job to run 1 at a time for example, is this possible?

mfrister commented 1 week ago

It looks like unique jobs go in the direction of what you want and I'm looking for something similar: A way to ensure a job doing an expensive operation can't have multiple concurrently running instances, so it doesn't put too much load on another system.

UniqueOpts with ByState look like they can almost do this. Explicitly setting the default values ensures there's no job inserted as long as there's another one in the DB, including completed jobs. Removing JobStateCompleted from the list sounds like it would do what what we want, but the documentation says (highlights by me):

The list may be safely customized to add additional states (cancelled or discarded), though only retryable may be safely removed from the list.

Warning: Removing any states from the default list (other than retryable) forces a fallback to a slower insertion path that takes an advisory lock and performs a look up before insertion. This path is deprecated and should be avoided if possible.

So I'm thinking maybe I shouldn't use this approach. Is there a better one?