Node are reported with status "Unknown" before being decommissioned

[AppliNH]

Description

Observed Behavior:

Since the upgrade to Karpenter 1.0.8, nodes being decommissioned are marked with an Unknown status for ~1min by the kube_node_status_condition Kubernetes metrics.

This wasn't the case before.

The exact PromQL we're using is count by (node) (kube_node_status_condition{status="unknown"} == 1) > 0

We've observed the following in the kubernetes-event-exporter logs for the nodes affected by this behavior:

• 2024-11-19 17:52:07.877 :

"reason":"InstanceTerminating",
"message":"Instance is terminating"

• 2024-11-19 17:52:42.101:

  "reason": "NodeNotReady",
  "message": "Node node-placeholder status is now: NodeNotReady",

• 2024-11-19 17:53:52.278:

"reason":"RemovingNode",
"message":"Node XXX event: Removing Node XXX from Controller"

Expected Behavior:

Nodes being decommissioned should not have Unknown status

Reproduction Steps (Please include YAML):

karpenter and karpenter-crd are deployed through Terraform:

module "addons" {
  source            = "aws-ia/eks-blueprints-addons/aws"
  version           = "v1.16.3"
  ...
  enable_karpenter = true
  karpenter = {
    chart_version = "1.0.8"
    values        = [templatefile("${path.module}/karpenter-values.yaml", {})]
  }
...
}

resource "helm_release" "karpenter_crd" {
  name       = "karpenter-crd"
  namespace  = "karpenter"
  repository = "oci://public.ecr.aws/karpenter"
  version    = "1.0.8"
  chart      = "karpenter-crd"
}

karpenter-values.yaml

tolerations:
- key: dedicated
  operator: Equal
  value: orchestration
  effect: NoSchedule
nodeSelector:
  purpose: orchestration
podLabels:
  part-of: karpenter
  team: sre
serviceMonitor:
  enabled: true
controller:
  resources:
    requests:
      cpu: 1
      memory: 3Gi
    limits:
     memory: 3Gi
settings:
  aws:
    defaultInstanceProfile: ...

Versions:

• Chart Version: 1.0.8

• Kubernetes Version (kubectl version): 1.30.4

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request

• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request

• If you are interested in working on this issue or have submitted a pull request, please leave a comment