custom label that does not use a restricted domain

[prashnttf]

Description

Observed Behavior: Deploying provsioners via spinnaker deployment custom labels doesn't support, Spinnaker does add additional labels during the chart deployment which is not supported currently on the karpenter crds

{"level":"ERROR","time":"2024-07-11T11:12:54.717Z","logger":"controller","message":"nodepool failed validation","commit":"490ef94","controller":"provisioner","NodePool":{"name":"victoriametrics"},"error":"invalid key name \"app.kubernetes.io/managed-by\": spec.template.metadata.labels\nlabel app.kubernetes.io/managed-by is restricted; specify a well known label: [karpenter.k8s.aws/instance-accelerator-count karpenter.k8s.aws/instance-accelerator-manufacturer karpenter.k8s.aws/instance-accelerator-name karpenter.k8s.aws/instance-category karpenter.k8s.aws/instance-cpu karpenter.k8s.aws/instance-cpu-manufacturer karpenter.k8s.aws/instance-ebs-bandwidth karpenter.k8s.aws/instance-encryption-in-transit-supported karpenter.k8s.aws/instance-family karpenter.k8s.aws/instance-generation karpenter.k8s.aws/instance-gpu-count karpenter.k8s.aws/instance-gpu-manufacturer karpenter.k8s.aws/instance-gpu-memory karpenter.k8s.aws/instance-gpu-name karpenter.k8s.aws/instance-hypervisor karpenter.k8s.aws/instance-local-nvme karpenter.k8s.aws/instance-memory karpenter.k8s.aws/instance-network-bandwidth karpenter.k8s.aws/instance-size karpenter.sh/capacity-type karpenter.sh/nodepool kubernetes.io/arch kubernetes.io/os node.kubernetes.io/instance-type node.kubernetes.io/windows-build topology.k8s.aws/zone-id topology.kubernetes.io/region topology.kubernetes.io/zone], or a custom label that does not use a restricted domain: [k8s.io karpenter.k8s.aws karpenter.sh kubernetes.io]\ninvalid key name \"app.kubernetes.io/name\": spec.template.metadata.labels\nlabel app.kubernetes.io/name is restricted; specify a well known label: [karpenter.k8s.aws/instance-accelerator-count karpenter.k8s.aws/instance-accelerator-manufacturer karpenter.k8s.aws/instance-accelerator-name karpenter.k8s.aws/instance-category karpenter.k8s.aws/instance-cpu karpenter.k8s.aws/instance-cpu-manufacturer karpenter.k8s.aws/instance-ebs-bandwidth karpenter.k8s.aws/instance-encryption-in-transit-supported karpenter.k8s.aws/instance-family karpenter.k8s.aws/instance-generation karpenter.k8s.aws/instance-gpu-count karpenter.k8s.aws/instance-gpu-manufacturer karpenter.k8s.aws/instance-gpu-memory karpenter.k8s.aws/instance-gpu-name karpenter.k8s.aws/instance-hypervisor karpenter.k8s.aws/instance-local-nvme karpenter.k8s.aws/instance-memory karpenter.k8s.aws/instance-network-bandwidth karpenter.k8s.aws/instance-size karpenter.sh/capacity-type karpenter.sh/nodepool kubernetes.io/arch kubernetes.io/os node.kubernetes.io/instance-type node.kubernetes.io/windows-build topology.k8s.aws/zone-id topology.kubernetes.io/region topology.kubernetes.io/zone], or a custom label that does not use a restricted domain: [k8s.io karpenter.k8s.aws karpenter.sh kubernetes.io]\ninvalid value: label app.kubernetes.io/managed-by is restricted; specify a well known label: [karpenter.k8s.aws/instance-accelerator-count karpenter.k8s.aws/instance-accelerator-manufacturer karpenter.k8s.aws/instance-accelerator-name karpenter.k8s.aws/instance-category karpenter.k8s.aws/instance-cpu karpenter.k8s.aws/instance-cpu-manufacturer karpenter.k8s.aws/instance-ebs-bandwidth karpenter.k8s.aws/instance-encryption-in-transit-supported karpenter.k8s.aws/instance-family karpenter.k8s.aws/instance-generation karpenter.k8s.aws/instance-gpu-count karpenter.k8s.aws/instance-gpu-manufacturer karpenter.k8s.aws/instance-gpu-memory karpenter.k8s.aws/instance-gpu-name karpenter.k8s.aws/instance-hypervisor karpenter.k8s.aws/instance-local-nvme karpenter.k8s.aws/instance-memory karpenter.k8s.aws/instance-network-bandwidth karpenter.k8s.aws/instance-size karpenter.sh/capacity-type karpenter.sh/nodepool kubernetes.io/arch kubernetes.io/os node.kubernetes.io/instance-type node.kubernetes.io/windows-build topology.k8s.aws/zone-id topology.kubernetes.io/region topology.kubernetes.io/zone], or a custom label that does not use a restricted domain: [k8s.io karpenter.k8s.aws karpenter.sh kubernetes.io]: spec.template.spec.requirements[5]\ninvalid value: label app.kubernetes.io/name is restricted; specify a well known label: [karpenter.k8s.aws/instance-accelerator-count karpenter.k8s.aws/instance-accelerator-manufacturer karpenter.k8s.aws/instance-accelerator-name karpenter.k8s.aws/instance-category karpenter.k8s.aws/instance-cpu karpenter.k8s.aws/instance-cpu-manufacturer karpenter.k8s.aws/instance-ebs-bandwidth karpenter.k8s.aws/instance-encryption-in-transit-supported karpenter.k8s.aws/instance-family karpenter.k8s.aws/instance-generation karpenter.k8s.aws/instance-gpu-count karpenter.k8s.aws/instance-gpu-manufacturer karpenter.k8s.aws/instance-gpu-memory karpenter.k8s.aws/instance-gpu-name karpenter.k8s.aws/instance-hypervisor karpenter.k8s.aws/instance-local-nvme karpenter.k8s.aws/instance-memory karpenter.k8s.aws/instance-network-bandwidth karpenter.k8s.aws/instance-size karpenter.sh/capacity-type karpenter.sh/nodepool kubernetes.io/arch kubernetes.io/os node.kubernetes.io/instance-type node.kubernetes.io/windows-build topology.k8s.aws/zone-id topology.kubernetes.io/region topology.kubernetes.io/zone], or a custom label that does not use a restricted domain: [k8s.io karpenter.k8s.aws karpenter.sh kubernetes.io]: spec.template.spec.requirements[6]"}

Expected Behavior: Label should be supported

Reproduction Steps (Please include YAML): apiVersion: karpenter.sh/v1beta1 kind: NodePool metadata: annotations: artifact.spinnaker.io/location: "" artifact.spinnaker.io/name: victoriametrics artifact.spinnaker.io/type: kubernetes/NodePool.karpenter.sh artifact.spinnaker.io/version: "" karpenter.sh/nodepool-hash: "5395453970923948961" karpenter.sh/nodepool-hash-version: v2 moniker.spinnaker.io/application: provisioners moniker.spinnaker.io/cluster: NodePool.karpenter.sh victoriametrics creationTimestamp: "2024-07-10T12:20:31Z" generation: 1 labels: app.kubernetes.io/managed-by: spinnaker app.kubernetes.io/name: provisioners name: victoriametrics resourceVersion: "216601977" uid: d27a749b-00b4-4325-ac32-a16efa19f418 spec: disruption: budgets:

• nodes: 10% consolidateAfter: 30m consolidationPolicy: WhenEmpty expireAfter: Never template: metadata: annotations: artifact.spinnaker.io/location: "" artifact.spinnaker.io/name: victoriametrics artifact.spinnaker.io/type: kubernetes/NodePool.karpenter.sh artifact.spinnaker.io/version: "" moniker.spinnaker.io/application: provisioners moniker.spinnaker.io/cluster: NodePool.karpenter.sh victoriametrics labels: app.kubernetes.io/managed-by: spinnaker app.kubernetes.io/name: provisioners role: victoriametrics-arm64 spec: nodeClassRef: name: arm64nodeclassgreen requirements:
  • key: kubernetes.io/arch operator: In values:
    • arm64
  • key: kubernetes.io/os operator: In values: Versions: 1.37
    • Chart Version:
    • Kubernetes Version (kubectl version): 1.27
    • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
    • Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
    • If you are interested in working on this issue or have submitted a pull request, please leave a comment

[rschalo]

Per https://github.com/kubernetes-sigs/karpenter/blob/main/pkg/apis/v1/labels.go#L62-L67, the label domain kubernetes.io is reserved and restricted for Karpenter's functionality. Can you share more about how Spinnaker needs these labels in order to function properly?

[prashnttf]

Per https://github.com/kubernetes-sigs/karpenter/blob/main/pkg/apis/v1/labels.go#L62-L67, the label domain kubernetes.io is reserved and restricted for Karpenter's functionality. Can you share more about how Spinnaker needs these labels in order to function properly?

https://spinnaker.io/docs/reference/providers/kubernetes-v2/#reserved-labels These are the labels which come under domain "kubernetes.io"

[YouhuaLi]

I have the same issue and I think the real issue is from Spinnaker. As you can see from the manifests:

  template:
    metadata:
      annotations:
        artifact.spinnaker.io/location: ""
        artifact.spinnaker.io/name: victoriametrics
        artifact.spinnaker.io/type: kubernetes/NodePool.karpenter.sh
        artifact.spinnaker.io/version: ""
        moniker.spinnaker.io/application: provisioners
        moniker.spinnaker.io/cluster: NodePool.karpenter.sh victoriametrics
      labels:
        app.kubernetes.io/managed-by: spinnaker
        app.kubernetes.io/name: provisioners

Spinnaker is adding some annotations and labels under .spec.template.metatdata but it should only do this at .metadata (from the yaml root).

So this looks like a Spinnaker bug.

[github-actions[bot]]

This issue has been inactive for 14 days. StaleBot will close this stale issue after 14 more days of inactivity.