search

collectiveidea / delayed_job

Database based asynchronous priority queue system -- Extracted from Shopify
http://groups.google.com/group/delayed_job
MIT License
4.81k stars 955 forks source link

Add support for worker pools that exclude specific queues #1148

Open njakobsen opened 2 years ago

njakobsen commented 2 years ago

Delayed Job already supports worker pools that draw from specific queues. This is useful to ensure those tasks are processed in a timely manner and do not starve. Coupling that with a pool that processes all jobs is a common practice. However, it is possible to create so many jobs for a specific that other jobs are delayed to an unacceptable degree. In this case we want to have a worker pool that only processes jobs that are not from that queue (or queues). There is currently no mechanism for this, hence this PR.

The changes allow worker pools to be named with a ! prefix which would indicate that we want this pool to process any jobs except those in the given queues. It is intended to be coupled with a change at the adapter level which would read the exclude_specified_queues flag and return the appropriate jobs based on whether the list of queues is intended as an inclusion or exclusion filter.

See this PR for changes made to DJ active_record adapter to support this new feature https://github.com/collectiveidea/delayed_job_active_record/pull/194.

feldpost commented 2 years ago

Any chance this one could be reviewed and merged?

dmytro-strukov commented 6 months ago

@albus522 could you please review this PR?