runabol / tork

A distributed workflow engine
https://tork.run
MIT License
509 stars 24 forks source link

Feature: task priority (RabbitMQ) #392

Closed runabol closed 1 month ago

runabol commented 1 month ago

Adds support for task priority when using RabbitMQ as a broker.

This features builds on RabbitMQ support for priority queues: https://www.rabbitmq.com/docs/priority

This change does require re-creating the task queues (by simply deleting them and letting Tork re-create them).

There is also a DB schema change adding the priority field to the tasks table:

alter table tasks add priority int default 0

To use priority in your jobs set the priority property on the task (values 1-9). Higher values indicate higher priority. Example:

- name: my first task
  image: alpine:3.18.3
  run: sleep 3
  priority: 1

This task will execute as a priority over tasks with no priority defined.

You can also set the default priority for a task at the job level:

name: my job
defaults:
  priority: 1
tasks:
- name: my first task
  image: alpine:3.18.3
  run: sleep 3

Workers that are already processing a task -- even a lower priority task -- will complete the task before considering any prioritized tasks.