collectiveidea / delayed_job

Database based asynchronous priority queue system -- Extracted from Shopify
http://groups.google.com/group/delayed_job
MIT License
4.81k stars 955 forks source link

Add support for worker pools that exclude specific queues [Green specs] #1198

Open dmytro-strukov opened 6 months ago

dmytro-strukov commented 6 months ago

Copy of https://github.com/collectiveidea/delayed_job/pull/1148, but with cleanup and green specs

Delayed Job already supports worker pools that draw from specific queues. This is useful to ensure those tasks are processed in a timely manner and do not starve. Coupling that with a pool that processes all jobs is a common practice. However, it is possible to create so many jobs for a specific that other jobs are delayed to an unacceptable degree. In this case we want to have a worker pool that only processes jobs that are not from that queue (or queues). There is currently no mechanism for this, hence this PR.

The changes allow worker pools to be named with a ! prefix which would indicate that we want this pool to process any jobs except those in the given queues. It is intended to be coupled with a change at the adapter level which would read the exclude_specified_queues flag and return the appropriate jobs based on whether the list of queues is intended as an inclusion or exclusion filter.

See this PR for changes made to DJ active_record adapter to support this new feature https://github.com/collectiveidea/delayed_job_active_record/pull/194.