aws / karpenter-provider-aws

Karpenter is a Kubernetes Node Autoscaler built for flexibility, performance, and simplicity.
https://karpenter.sh
Apache License 2.0
6.68k stars 934 forks source link

custom resource nodepools.karpenter.sh isn't configured for webhook conversion #7124

Closed mijndert closed 2 days ago

mijndert commented 2 days ago

Description

Observed Behavior:

Seeing a lot of errors like in the title after upgrading Karpenter from 0.34.6 to 0.37.3 on EKS 1.30.

The documentation isn't incredibly clear on these types of errors. I'm kind of lost on what to do here. It's probably something I missed during the upgrade. All I did was change the chart version just like previous upgrades.

Expected Behavior:

Reproduction Steps (Please include YAML):

  1. Upgrade to Karpenter 0.37.3 from 0.34.6
resource "helm_release" "karpenter" {
  namespace        = "karpenter"
  create_namespace = true

  name                = "karpenter"
  repository          = "oci://public.ecr.aws/karpenter"
  repository_username = data.aws_ecrpublic_authorization_token.token.user_name
  repository_password = data.aws_ecrpublic_authorization_token.token.password
  chart               = "karpenter"
  version             = "v0.37.3"

  values = [
    <<-EOT
    settings:
      clusterName: ${module.eks.cluster_name}
      clusterEndpoint: ${module.eks.cluster_endpoint}
      interruptionQueueName: ${module.karpenter.queue_name}
    serviceAccount:
      annotations:
        eks.amazonaws.com/role-arn: ${module.karpenter.irsa_arn} 
    controller:
      resources:
        requests:
          cpu: 1
          memory: 1Gi
        limits:
          cpu: 1
          memory: 1Gi
    EOT
  ]
}

resource "kubectl_manifest" "karpenter_node_pool_default" {
  yaml_body = <<-YAML
    apiVersion: karpenter.sh/v1beta1
    kind: NodePool
    metadata:
      name: default-nodepool
    spec:
      template:
        metadata:
          labels:
            nodegroup: default
        spec:
          nodeClassRef:
            name: default
          requirements:
            - key: node.kubernetes.io/instance-type
              operator: In
              values: ${jsonencode(var.default_nodepool_instance_type)}
            - key: "karpenter.sh/capacity-type"
              operator: In
              values: ${jsonencode(var.default_nodepool_capacity)}
      disruption:
        consolidationPolicy: WhenUnderutilized
        expireAfter: 720h
  YAML

  depends_on = [
    kubectl_manifest.karpenter_node_class
  ]
}

Versions:

mijndert commented 2 days ago

Upgrading the CRD's fixed the crashing but resulted in a host of new errors about the webhook wanting to upgrade to v1.