Make use of queue priorities

[mzur]

Configure and use different queue priorities. Maybe high, default, and low.

Candidates for low might be very long running tasks like:

• Generating image thumbnails
• Processing a laserpoint detection

Go over all modules and their queued jobs and think about which priority might fit.

Finally we can run two queue workers on our Cebitec instance. One for the high and default, the other for the default and low priorities.

This will mitigate job congestion like we experienced when e.g. lots of laserpoint detection jobs were submitted.

[mzur]

This is as easy as changing the queue for each submission of a job like this

dispatch((new Job)->onQueue('high'));

However before we implement this, we need to start multiple queue workers who listen to the different queues, too. This adds complexity to any on-premise installation of Biigle and might not be very important for the load that we are currently expecting. A single queue will probably do fine since additional queue workers can be added at any time. Closing this in favor of simplicity.

[mzur]

Reopening because we should consider this after all. We repeatedly had massively delayed jobs to generate reports because some other things (like generating lots of annotation patches) clogged the queue.

Make a high priority queue. Change the config value queue.connections.database.queue to high,default. This will make all workers process jobs on the high queue first.

[mzur]

Changing this to high because I found a trivial way to implement this (see previous post) which won't add any complexity to existing and future installations.

[mzur]

The "trivial way" does not work, unfortunately. Setting the queue to high,default makes the queue workers act correctly but new jobs that are submitted to the default queue get the queue high,default, too, instead of the correct default.