Proposal: Specifying priority classes for Hypershift control plane workloads

[csrwng]

Control plane workloads on a hypershift management cluster should take precedence over other lower priority workloads. Furthermore, within control plane workloads, etcd is special because it is not stateless in the same way that other pods are in the control plane (ie. kube apiserver, kube controller manager, etc)

In order to avoid early eviction of these workloads due to cpu/memory/disk pressure on nodes, we should specify PriorityClassNames for pods belonging to a hypershift control plane (see https://github.com/openshift/hypershift/issues/238).

We can define 2 different PriorityClass(es):

• HypershiftControlPlane
• HypershiftEtcd

The hypershift operator can take the responsibility of creating these classes if they don't exist. All control plane components can be assigned the HypershiftControlPlane class with the exception of etcd pods which would et the HypershiftEtcd class. Service providers can create/update these classes as they see fit to adapt to their use case. As long as the priority classes exist, the hypershift operator will not update their spec.

Todo:

• [ ] Figure out reasonable priority defaults for these 2 classes
• [ ] Determine if there's any workload in the control plane that should not be assigned these classes.

[ironcladlou]

Tracked here: https://issues.redhat.com/browse/HOSTEDCP-191