search

kubernetes / autoscaler

Autoscaling components for Kubernetes
Apache License 2.0
7.8k stars 3.87k forks source link

Cluster Autoscaler: align workload-level APIs with Karpenter #6648

Open towca opened 3 months ago

towca commented 3 months ago

Which component are you using?: Cluster Autoscaler

Is your feature request designed to solve a problem? If so describe the problem this feature should solve.:

Recently, Karpenter officially joined sig-autoscaling, and we now have 2 Node autoscalers officially supported by Kubernetes. Both autoscalers provide workload-level APIs that a workload owner can use to influence autoscaling behavior related to the workloads. Some of these APIs have identical semantics, but different naming. Because of this, workloads taking advantage of such APIs aren't portable between clusters using different autoscalers (e.g. in a multi-cloud setting).

Cluster Autoscaler provides the following workload-level APIs:

To my knowledge, right now Karpenter only provides the following workload-level API:

Describe the solution you'd like.:

  1. Introduce a new API prefix for concepts related specifically to Node autoscaling: node-autoscaling.kubernetes.io. Going forward, any new APIs using this prefix would have to be approved by both CA and Karpenter owners. Note that this doesn't prevent the autoscalers from adding new autoscaler-specific APIs, but the goal should be to use the common prefix if possible.
  2. Add support for node-autoscaling.kubernetes.io/do-not-disrupt: true to CA and Karpenter, while still honoring cluster-autoscaler.kubernetes.io/safe-to-evict: false and karpenter.sh/do-not-disrupt: true for backwards compatibility.
  3. Ideally we'd also add support for node-autoscaling.kubernetes.io/do-not-disrupt: false, mapping to safe-to-evict: true in CA. Not sure what the semantics for that in Karpenter would be (need to check if it has any consolidation-blocking conditions triggered by a pod).
  4. Align with Karpenter on if they're interested in implementing any of the other workload-level APIs that CA uses, and if so - migrate them to the common API prefix as well.

Describe any alternative solutions you've considered.:

The CA/Karpenter alignment AEP also mentions aligning on Node-level APIs related to scale-down/consolidation. However, the scope of these APIs will likely be Node lifecycle altogether, not just Node autoscaling. IMO we shouldn't mix the two API prefixes together, and the Node-level API migration should be handled separately. Taking do-not-disrupt: true as an example: if we put it in a node-autoscaling.kubernetes.io prefix, all we need to guarantee is that the 2 supported autoscalers handle it correctly. If we were to put it into a broader node-lifecycle.kubernetes.io prefix, every component interacting with node lifecycle through this API would have to honor it going forward, or break the expectations. Honoring do-not-disrupt: true might not be an option for certain components (e.g. a component Upgrading nodes with strict FedRAMP requirements has to violate it at some point), limiting the usefulness of that broader node-lifecycle API.

Additional context.:

towca commented 3 months ago

@MaciekPytel @gjtempleton @jonathan-innis I want to discuss this during the next sig meeting if possible, could you take a look?

sftim commented 3 months ago

BTW, if you're looking at the key prefix for annotations and / or labels, these things aren't called “API groups”. We use the term API group purely for resource kinds that you find within Kubernetes' HTTP API.

sftim commented 3 months ago

This feels accurate: /retitle Align annotations and labels between Cluster Autoscaler and Karpenter

towca commented 3 months ago

Good point about the "API group" being a precisely defined term, changed to "API prefix". Unless we have a name for such concept as well?

I'm struggling to understand how the new title is accurate. "Aligning labels and annotations" could mean many things in the Cluster Autoscaler/Karpenter context, since labels and annotations are important for various parts of the logic (e.g. something around node templates would probably be my first guess for something related to "aligning labels and annotations"). "Workload-level APIs", on the other hand, should be pretty clear in Cluster Autoscaler/Karpenter context.

/retitle Cluster Autoscaler: align workload-level APIs with Karpenter

sftim commented 3 months ago

If you mean the cluster-autoscaler.kubernetes.io in cluster-autoscaler.kubernetes.io/safe-to-evict-local-volumes @towca, I tend to call that a label name prefix or annotation name prefix. See https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set for some more detail.

elmiko commented 3 months ago

thanks for bringing this up @towca , i think that if we make an api around this it will be beneficial to the wider community. i don't have super strong opinions on the naming part, but i think de-emphasizing the "autoscaling" part of it would be nice. that said, i like the distinction you call about what would be expected from something with the node-lifecycle as opposed to node-autoscaling part of its prefix.i would have thought node-lifecycle would be a little better, but i like your point about other lifecycle tooling having to then obey them.

k8s-triage-robot commented 1 week ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

towca commented 6 days ago

The https://github.com/kubernetes/kubernetes/pull/124800 PR defining the first common annotation is in review. The review has stalled a bit, I bumped it for the reviewers during the sig-autoscaling meeting today.

towca commented 6 days ago

/remove-lifecycle stale