Add option to jms-job-queue:run command to run jobs from specific queue

[ghost]

A new option should be added to jms-job-queue:run command to allow different workers handle different tasks (queues). For example you could have a worker only for downloading images, another one for handling other tasks etc.

Option should accept many queue names such as queue1,queue2 in case we need to consume more than one queues in that worker.

Issue #76 could benefit from this as well.

[nclavaud]

+1

I've been thinking about horizontal scaling and came up with the same idea. I think this is definitely an easy way to run several workers at the same time.

[mkocztorz]

This is very interesting feature. Would you consider merging it?

The use case is this: we have centralized DB and multiple server nodes each would be running a worker. Each node can add jobs to any queue but there are specialized nodes that can actually do the jobs.

[rcousens]

I really like this idea as well and am considering implementing it myself.

The goal would be to enable some degree of horizontal scalability by limiting an instance of the run command to processing jobs from a particular queue.

The use case I have in mind is a server instance that is dedicated to processing print jobs and reports (generally PDF conversion, manipulation and aggregation, trivial data crunching/statistics etc), while another server instance would be dedicated to processing uploads, OCR and image resizing. By locking an instance of the run command to a particular queue we can avoid race conditions but dedicate the entire available system resources to dealing with a distinct queue.

Currently, even with minimal concurrency, some jobs compete for available resources and slow down the execution of other jobs that may be higher priority.

Having more fine grained control means we can match servers and specifications to the tasks they are intended to process and avoid contention for CPU and memory between tasks at different priority levels. Essentially, this change would enable low priority resource intensive tasks to process independently of higher priority tasks (which often demand quick feedback to users) and facilitate an overall higher maximum throughput for both queues by avoiding competition for available resources.

The downside is, depending on usage, some server instances may sit idle. This is a penalty we are prepared to incur for the flexibility of ensuring a particular run instance is always available to process tasks of a given queue.

@schmittjoh Your thoughts?

[rcousens]

This feature is now in the bundle and as such this ticket can be closed.

@schmittjoh

[schmittjoh]

Yep, thanks!