search

kubernetes-sigs / karpenter

Karpenter is a Kubernetes Node Autoscaler built for flexibility, performance, and simplicity.
Apache License 2.0
540 stars 181 forks source link

karpenter does not add `karpenter.sh/capacity-type` label to the nodes #660

Closed tmoreadobe closed 10 months ago

tmoreadobe commented 10 months ago

Description

Observed Behavior: For the nodes which is karpenter spinning up, karpenter is not adding karpenter.sh/capacity-type as both machine and provisioner has this label. Expected Behavior: karpenter should add the expected label for consolidation to work. Reproduction Steps (Please include YAML): provisionerspec

Name:         core-karpenter-ethos-core-karpenter-worker2
Namespace:    
Labels:       adobe.com/appname=karpenter-ethos101-dev-va6
              app.kubernetes.io/instance=core-karpenter
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ethos-core-karpenter
              app.kubernetes.io/version=v0.31.0
              ethos.adobe.net/managed-by=k8s-infrastructure.helm
              helm.sh/chart=ethos-core-karpenter-0.0.5
Annotations:  karpenter.sh/provisioner-hash: 1417663675986826836
API Version:  karpenter.sh/v1alpha5
Kind:         Provisioner
Metadata:
  Creation Timestamp:  2023-10-28T00:44:25Z
  Generation:          2
  Resource Version:    2306126405
  UID:                 6a302235-6c20-4a36-aa03-1c31caa7c7f3
Spec:
  Consolidation:
    Enabled:  true
  Labels:
    ethos.adobe.net/node-templateVersion:     a8a12c5ee31a4668f93b01ccf92ebf7c54679f693bd3a14b4bd86086432388
    kubernetes.io/os:                         linux
    node.kubernetes.io/ethos-workload.amd64:  true
    node.kubernetes.io/node-group:            worker2-group
    node.kubernetes.io/role:                  worker
  Limits:
    Resources:
      Cpu:     1k
      Memory:  1000Gi
  Provider Ref:
    Name:  core-karpenter-ethos-core-karpenter-worker2
  Requirements:
    Key:       karpenter.k8s.aws/instance-family
    Operator:  In
    Values:
      c5
      c6g
      m5
      m6g
      r5
      r6g
    Key:       karpenter.k8s.aws/instance-generation
    Operator:  Gt
    Values:
      4
    Key:       kubernetes.io/arch
    Operator:  In
    Values:
      amd64
    Key:       karpenter.sh/capacity-type
    Operator:  In
    Values:
      on-demand
    Key:       topology.kubernetes.io/zone
    Operator:  In
    Values:
      us-east-1a
      us-east-1b
      us-east-1c
    Key:       kubernetes.io/os
    Operator:  In
    Values:
      linux
Status:
  Resources:
    Attachable - Volumes - Aws - Ebs:  25
    Cpu:                               16
    Ephemeral - Storage:               194314260Ki
    Memory:                            32054796Ki
    Pods:                              112
Events:                                <none>

machinespec

Name:         core-karpenter-ethos-core-karpenter-worker2-zjxnb
Namespace:    
Labels:       ethos.adobe.net/node-templateVersion=a8a12c5ee31a4668f93b01ccf92ebf7c54679f693bd3a14b4bd86086432388
              karpenter.k8s.aws/instance-category=c
              karpenter.k8s.aws/instance-cpu=72
              karpenter.k8s.aws/instance-encryption-in-transit-supported=false
              karpenter.k8s.aws/instance-family=c5
              karpenter.k8s.aws/instance-generation=5
              karpenter.k8s.aws/instance-hypervisor=nitro
              karpenter.k8s.aws/instance-memory=147456
              karpenter.k8s.aws/instance-network-bandwidth=25000
              karpenter.k8s.aws/instance-pods=737
              karpenter.k8s.aws/instance-size=18xlarge
              karpenter.sh/capacity-type=on-demand
              karpenter.sh/provisioner-name=core-karpenter-ethos-core-karpenter-worker2
              kubernetes.io/arch=amd64
              kubernetes.io/os=linux
              node.kubernetes.io/ethos-workload.amd64=true
              node.kubernetes.io/instance-type=c5.18xlarge
              node.kubernetes.io/node-group=worker2-group
              node.kubernetes.io/role=worker
              topology.kubernetes.io/region=us-east-1
              topology.kubernetes.io/zone=us-east-1a
Annotations:  karpenter.k8s.aws/nodetemplate-hash: 7594172995892393789
              karpenter.sh/managed-by: ethos101-dev-va6-k8s-eks-master
              karpenter.sh/provisioner-hash: 1417663675986826836
API Version:  karpenter.sh/v1alpha5
Kind:         Machine
Metadata:
  Creation Timestamp:  2023-10-30T19:02:52Z
  Finalizers:
    karpenter.sh/termination
  Generate Name:  core-karpenter-ethos-core-karpenter-worker2-
  Generation:     1
  Owner References:
    API Version:           karpenter.sh/v1alpha5
    Block Owner Deletion:  true
    Kind:                  Provisioner
    Name:                  core-karpenter-ethos-core-karpenter-worker2
    UID:                   6a302235-6c20-4a36-aa03-1c31caa7c7f3
  Resource Version:        2306109951
  UID:                     6b20ee72-270f-428e-98e4-dad254147d58
Spec:
  Machine Template Ref:
    Name:  core-karpenter-ethos-core-karpenter-worker2
  Requirements:
    Key:       karpenter.sh/provisioner-name
    Operator:  In
    Values:
      core-karpenter-ethos-core-karpenter-worker2
    Key:       ethos.adobe.net/node-templateVersion
    Operator:  In
    Values:
      a8a12c5ee31a4668f93b01ccf92ebf7c54679f693bd3a14b4bd86086432388
    Key:       karpenter.sh/capacity-type
    Operator:  In
    Values:
      on-demand
    Key:       topology.kubernetes.io/zone
    Operator:  In
    Values:
      us-east-1a
      us-east-1b
    Key:       node.kubernetes.io/ethos-workload.amd64
    Operator:  In
    Values:
      true
    Key:       node.kubernetes.io/role
    Operator:  In
    Values:
      worker
    Key:       karpenter.k8s.aws/instance-generation
    Operator:  Gt
    Values:
      4
    Key:       kubernetes.io/arch
    Operator:  In
    Values:
      amd64
    Key:       kubernetes.io/os
    Operator:  In
    Values:
      linux
    Key:       node.kubernetes.io/instance-type
    Operator:  In
    Values:
      c5.12xlarge
      c5.18xlarge
      c5.24xlarge
      c5.4xlarge
      c5.9xlarge
      c5.metal
      m5.12xlarge
      m5.16xlarge
      m5.24xlarge
      m5.4xlarge
      m5.8xlarge
      m5.metal
      r5.12xlarge
      r5.16xlarge
      r5.24xlarge
      r5.4xlarge
      r5.8xlarge
      r5.metal
    Key:       node.kubernetes.io/node-group
    Operator:  In
    Values:
      worker2-group
    Key:       karpenter.k8s.aws/instance-family
    Operator:  In
    Values:
      c5
      c6g
      m5
      m6g
      r5
      r6g
  Resources:
    Requests:
      Cpu:     10237m
      Memory:  23845317289
      Pods:    19
Status:
  Allocatable:
    Cpu:                  71750m
    Ephemeral - Storage:  179Gi
    Memory:               127934Mi
    Pods:                 737
  Capacity:
    Cpu:                  72
    Ephemeral - Storage:  200Gi
    Memory:               136396Mi
    Pods:                 737
  Conditions:
    Last Transition Time:  2023-10-30T19:04:45Z
    Status:                True
    Type:                  MachineInitialized
    Last Transition Time:  2023-10-30T19:02:55Z
    Status:                True
    Type:                  MachineLaunched
    Last Transition Time:  2023-10-30T19:04:45Z
    Status:                True
    Type:                  Ready
    Last Transition Time:  2023-10-30T19:04:13Z
    Status:                True
    Type:                  MachineRegistered
  Node Name:               ip-10-95-64-98.ec2.internal
  Provider ID:             aws:///us-east-1a/i-0fc6c888b14b25ac9
Events:
  Type    Reason                 Age                  From       Message
  ----    ------                 ----                 ----       -------
  Normal  DeprovisioningBlocked  30s (x73 over 152m)  karpenter  Cannot deprovision Machine: Required label "karpenter.sh/capacity-type" doesn't exist

error on the node same as of machine Versions:

jonathan-innis commented 8 months ago

Yes let me add it here what label maker is doing

After reading through the shared code, I realize that there is a third scenario:

  1. You have a client that is fully overwriting the section when doing an UPDATE operation rather than just updating the parts that it specifically cares about.

In this case, I can see that Label Maker is retrying on conflict (which means that it respects the conflict response if the resource version is out of date); however, when it does the retry, it looks like it still overwrites the labels based on the old version of the labels from the initial GET operation here: https://github.com/tmoreadobe/label-maker/blob/master/pkg/controller/labelmaker/labelmaker_controller.go#L95. If you wanted to make sure that this was properly handled, you would basically need to make this callback function here https://github.com/tmoreadobe/label-maker/blob/master/pkg/controller/labelmaker/labelmaker_controller.go#L129 only set the specific labels that you care about rather than fully overwriting the labels with the outdated version.

Can you explain in more depth what label maker is doing and why you have a need for it for your use-case? Is there anything that we can leverage within Karpenter or elsewhere to enable this label updating if the component there is out-of support?

tmoreadobe commented 7 months ago

yeah understood @jonathan-innis . I already have made changes and this should work but the problem with this app is that it's very old and the build process is busted. Is there a possibility that we can add custom labels in the startup on all the nodes provisioned by karpenter?

tmoreadobe commented 7 months ago

The label-maker is adding node-role.kubernetes.io/node to the worker nodes. When we list nodes, the roles can be seen with this label. We have certain monitoring usecases where we need this label. If this can be supported within karpenter, this would resolve lot of the hassle we are going through and make us adopt karpenter fairly quickly.

jonathan-innis commented 7 months ago

The label-maker is adding node-role.kubernetes.io/node to the worker nodes

Just so I understand the challenge here: Is it that you need the value of that label to be templated based on the name of the node? Is that what's restricting you from just being able to add that label directly to the NodePool?

tmoreadobe commented 7 months ago

Yes we need that label on the worker nodes but the issue is we dont use nodepools/nodegroups. If we use nodepools, we can configure those labels in the nodepools itself in EKS.

jonathan-innis commented 7 months ago

Yes we need that label on the worker nodes but the issue is we dont use nodepools/nodegroups

NodePool has a label section, does that not work for you? https://karpenter.sh/docs/concepts/nodepools/#:~:text=to%20all%20nodes-,labels,-%3A%0A%20%20%20%20%20%20%20%20billing%2Dteam

(I'm also realizing you may not be on the latest version of Karpenter yet so NodePool == Provisioner)

tmoreadobe commented 7 months ago

Okay, are these labels applied as soon as the node joins the cluster.

And yes we are on 0.27.3 version of karpenter since we were facing this issue post 0.28 releases šŸ˜…

tmoreadobe commented 7 months ago

@jonathan-innis this label for node-role is restricted and is not getting applied when specified in the label section. Do you know of any other way a karpenter can accomodate this based on the node.kubernetes.io/role: worker label. I see this issue but it is closed and valid use case for many of the users. https://github.com/aws/karpenter-provider-aws/issues/1941

sftim commented 6 months ago

For a comment about node-role.kubernetes.io/worker, see https://github.com/kubernetes-sigs/karpenter/issues/1046#issuecomment-1973571758