Add NodePoolGroupLimit CRD for limits that span NodePools

Description

What problem are you trying to solve?

Today, limits can only be specified on individual NodePools. While this is fine for simple situations, it is insufficient when multiple NodePools comprise a logical grouping of compute that should share a limit. This happens most often when there are important variations in a NodePool's properties beyond its requirements that mandate the use of multiple NodePools, but when they are otherwise related in some way relevant to limits (e.g. same department, team, application, budget line-item).

For example, an organization might group limits by team. A team might require nodes labelled in two distinct ways, necessitating the use of two NodePools. Splitting the team's limit in half for each NodePool might not be sufficient if the balance of nodes between the NodePools varies over time.

I propose a NodePoolGroupLimit CRD (or a similar appropriate name) that would allow a defined limit to apply to NodePools chosen by a selector. If multiple NodePoolGroupLimit objects select the same NodePool, the most prohibitive limit should take precedence.

It might look something like this:

apiVersion: karpenter.sh/v1
kind: NodePoolGroupLimit
metadata:
  name: frontend-bingo
  labels:
    team: frontend
    service: bingo
spec:
  selector:
    # label selector for NodePool labels
    team: frontend
    service: bingo
  limits:
    cpu: "100"
    memory: 128Gi

How important is this feature to you?

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

kubernetes-sigs / karpenter

Add NodePoolGroupLimit CRD for limits that span NodePools #1747

Description