Feature Request: Task Priority

[pavdwest]

Overview

Are there any plans to support task priorities in the future?

Common use cases are prioritisation via queues with different priorities or directly at task enqueue time. I've also seen prioritisation via Task definition which is a bit too rigid in my opinion and can easily be achieved by other means if either of the first two options are available.

Note that the assumption here is that the tasks are processed by a single group of homogeneous workers (as opposed to e.g. multiple groups each processing specific queues).

Queue Priorities

e.g.

high_priority_queue = Queue(priority=5)
medium_priority_queue = Queue(priority=50)
low_priority_queue = Queue(priority=100)

t1 = Task(queue=low_priority_queue)
t2 = Task(queue=medium_priority_queue)
t3 = Task(queue=high_priority_queue)

tasks = [t1, t2, t3]

SomeTaskFramework.enqueue_tasks(tasks)

Task Priorities (at enqueue time)

e.g.

t1 = Task(priority=100)
t2 = Task(priority=50)
t3 = Task(priority=5)

tasks = [t1, t2, t3]

SomeTaskFramework.enqueue_tasks(tasks)

Expectation

The expectation is that the tasks will complete in the order t3, t2, t1 for both cases.

[JonasKs]

This would essentially only be a matter of which queue to pick from first, not which task to process, right?

So if your worker is configured to max_jobs=10 jobs and you have 100 jobs in each queue, it will pick 10 jobs from the priority queue 10 times before moving on to the next queue?
If there's 7 tasks in the priority queue and 10 in the next queue, it would pick 7 from the priority queue and 3 from the next queue?