[EKS] [request]: Using Karpenter in a multi-tenancy EKS Cluster - Need to assign Label to Worker nodes with namespace name

[mmasaaud]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request What do you want us to build?

• We planning to use Karpenter for autoscaling in our EKS clusters, this is a multi-tenant EKS clusters and each tenant should have the ability to choose and create there own worker nodes based on their own workload requirements.
• As a platform team we need to know which worker nodes realted to which team so we need to enforce kind or labeling to worker nodes on Karpenter and EKS scheduler to enforce this labeling strategy for each Name Space ( tenant ).
• Using OPA GK or Kybverno is not really efficient ion this situation today in our environment, enforcing multi-tenancy with the OPA brings a lot of problems on our side, basically we will have to think of a lot of integrations and policing. Therefore Karpenter will be narrowed to few or little use-cases/clients within our org.
• We are looking to have Karpenter and EKS Scheduler to be more multi-tenant friendly by having at least Provisioner CRD has to have a namespace.

• Which service(s) is this request for? EKS, Karpenter

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? What outcome are you trying to achieve, ultimately, and why is it hard/impossible to do right now? What is the impact of not having this problem solved? The more details you can provide, the better we'll be able to understand and solve the problem.

• We looking to have Karpenter more Multi-tenant friendly by having a namespace spec on it's CRDs so we can know which namespace those worker nodes related to.
• We will not be able to use Karpenter in our environment due to this missing capabilities.

Are you currently working around this issue? How are you currently solving this problem?

• Using OPA GK to Mutate Pods with Toleration and NodeSelector and assign labels to the newly created nodes whenever Karpenter add them.

This is not really working on a large scale!! Additional context Anything else we should know?

This is blocking us on Certifying Karpenter within our environment. Attachments If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)

[dims]

cc @ellistarn

[ellistarn]

This may be a better question for: https://github.com/aws/karpenter/issues, but happy to answer here,

We are looking to have Karpenter and EKS Scheduler to be more multi-tenant friendly by having at least Provisioner CRD has to have a namespace.

Provisioners can't be scoped to a namespace using Karpenter specific concepts. If Karpenter were to implement something like this, the kube scheduler would not enforce its decisions once the nodes come online. Instead, this must be implemented using Kubernetes native scheduling rules (i.e. taints, nodeaffinity)

As you've identified, once way to separate workloads with Kubernetes native scheduling rules is to use node selectors to ensure that different tenants run on different nodes. To repel workloads that don't specify nodeAffinity, you can use a taint and corresponding pod toleration to protect against this.

However, this places a burden on service teams to apply the correct node affinity for their workloads. If they don't they may not schedule, or worse, may schedule onto spare multi-tenant capacity. One way to solve this (as you've identified) is with a policy enforcement system like OPA or Kyverno. These systems use mutating webhooks to inject things like nodeAffinity into pods. However, you need this nodeAffinity to be dynamic, according to namespace.

Using OPA GK or Kybverno is not really efficient ion this situation today in our environment, enforcing multi-tenancy with the OPA brings a lot of problems on our side, basically we will have to think of a lot of integrations and policing. Therefore Karpenter will be narrowed to few or little use-cases/clients within our org.

As written, it's not clear to me what problems you're facing with existing policy manager. If I were to guess, I'd assume that it's not possible to easily express namespace-based nodeaffinity rules with existing policy managers.

We have a few options:

1. Build this into upstream k8s (long road)
2. Build this into Karpenter (e.g. https://github.com/aws/karpenter-core/issues/74)
3. Build this as a new 3p project (e.g. github.com/kubernetes-sigs/tenancy-webhook)

I've played around with #3 in the past, and was able to hack up a solution in about ~100 lines of code. I'd be happy to provide some pointers if this is something you would be interested in taking on. #2 is possible, but I worry a little bit about expanding Karpenter's scope to include pod webhooks -- this type of policy enforcement seems like a better fit for existing policy enforcement solutions.

As a final note, you didn't mention this in your post, so it's possible you didn't run into this yet, but if you can get something to inject the scheduling rules, you can achieve pod isolation using a single Karpenter provisioner and the exists operator: https://karpenter.sh/v0.25.0/concepts/scheduling/#exists-operator.