Please clarify Armada Job Priority vs Priority Class vs Queue priorityFactor

[PhilC1234]

It is not clear how these three features work and what is the relationship among them.

In Armada docs, "priority: the job priority (lower values indicate higher priority)." And for Kubernetes priorityclass, "The higher the value, the higher the priority. "

This is confusing, maybe it will be good the definition of higher priority can be consistent.

Also what is the usage of Queue priorityFactor? how to set the value?

In addition, when I tried to set the priority for the jobs with using default priorityclass, the priority seems not working as defined.

[dejanzele]

Hi @PhilC1234,

The priority field is used to control ordering within the queue in Armada side. PriorityClass is used to control preemption of jobs between users.

Parts of the Scheduler docs:

Jobs are totally ordered within each queue by:

Priority class priority.
Job priority.
Time submitted.

Docs on preemption:

Armada supports two forms of preemption:

Urgency-based preemption, i.e., making room for a job by preempting less urgent jobs. This form of preemption works in the same way as the normal Kubernetes preemption.
Preemption to fair share, i.e., preempting jobs belonging to users with more than their fair share of resources, such that those resources can be re-allocated to improve fairness.
Both forms of preemption are based on Armada priority classes (PCs). These are similar to but distinct from Kubernetes PCs. All Armada jobs have an Armada PC associated with them. Each Armada PC is represented by the following fields:

name: A unique name associated with each Armada PC.
priority: An integer encoding the urgency of jobs with this PC. Jobs with a PC with higher priority can always preempt jobs with a PC with lower priority.
isFairSharePreemptible: A boolean indicating whether jobs with this PC can be preempted via preemption to fair share. Note that all jobs can be preempted via urgency-based preemption, unless there is no other job with a higher PC priority.
Job priority classes are set by setting the priorityClassName field of the podspec embedded in the job. Jobs with no PC are automatically assigned one. We describe both forms of preemption in more detail below.

Please check the Scheduler docs for more info - https://github.com/armadaproject/armada/blob/master/docs/scheduler.md

[dejanzele]

@PhilC1234

In latest Armada version, preemption is enabled by default and configured in the Scheduler - https://github.com/armadaproject/armada/blob/master/config/scheduler/config.yaml#L92-L101

The defined PriorityClass objects in this config must match the priority and be created in all executor clusters.

Also in the config you can specify which class can be preempted.