search

kubernetes-sigs / karpenter

Karpenter is a Kubernetes Node Autoscaler built for flexibility, performance, and simplicity.
Apache License 2.0
630 stars 205 forks source link

Add the possibility to use "do-not-disrupt" on namespace level #1189

Open dschunack opened 7 months ago

dschunack commented 7 months ago

Description

Description

What problem are you trying to solve? We have hundert's of Jenkins Masters and Clients running in dedicated namespaces and it's hard to add the "do-not-disrupt" annotation on all the PODs for Master, Clients and Schedules Jobs. The possibility to set it on namespace level would help a lot for tools they have only a single POD like Jenkins.

How important is this feature to you? It's important for us to reduce the overhead of manage the annotation for all the Deployments, STS, Jobs and PODs. With this feature would it be very simple.

Related to Issue 6028

jonathan-innis commented 6 months ago

it's hard to add the "do-not-disrupt" annotation on all the PODs for Master, Clients and Schedules Jobs

I'm curious what makes setting the annotations on all of the pods in a given namespace difficult. Is there some limitation around what annotations you can set on pods from a given vendor or is just that you have app teams that have control over these pods and you aren't always guaranteed to get an annotation on all of the pods? Is this something that you could obtain through a policy agent or some custom mutating webhook that could update all pods in a given namespace to always have this annotation rather than passing this responsibility down to Karpenter? I'm curious what you see as the advantages of having it at one layer vs. the other.

dschunack commented 6 months ago

Hi,

It's not a limitation around the annotation, it's more the second one. We Provide the Kubernetes Platform. Based on this our Customers provide a CI/CD Jenkins Service to there Customers. That means, around 150 Jenkins Master, 800 Pipelines and thousands of Builds per Day of thousands of engineers. It's not easy to guarantee that all Build Pods have the right annotations. If we start to talk with all the engineers it takes a long time without a guarantee that we reach all.

Mutating could be an option, but for the build eks we need to rollout kyverno and we need to write and maintain the mutation policies. If Kyverno doesn't work due to a bug/error, nothing works anymore and all to inject an annotation. We had this in the past and this is not funny :-)

In our opinion, the easiest way is to set an annotation at the namespace level to reach all customers without much effort.

jonathan-innis commented 6 months ago

If Kyverno doesn't work due to a bug/error, nothing works anymore and all to inject an annotation. We had this in the past and this is not funny

I wonder if the MutatingAdmissionPolicy is the right solve for this long-term. This would introduce adding this kind of mutation as a feature of Kubernetes, rather than causing you to have to rely on a separate component.

The other thing is if you don't want to take on the full dependency closure of Kyverno, I could see you creating your own mutating webhook or your own controller to add this functionality.

k8s-triage-robot commented 3 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

dschunack commented 2 months ago

/remove-lifecycle stale