Karpenter always launches c6a instance regardless of provisioner configuration

byndcivilization commented 1 year ago

Version

Karpenter Version: v0.26.1

Kubernetes Version: v1.25.6-eks-48e63af

$kc version --output=yaml
clientVersion:
  buildDate: "2023-01-18T15:58:16Z"
  compiler: gc
  gitCommit: 8f94681cd294aa8cfd3407b8191f6c70214973a4
  gitTreeState: clean
  gitVersion: v1.26.1
  goVersion: go1.19.5
  major: "1"
  minor: "26"
  platform: darwin/arm64
kustomizeVersion: v4.5.7
serverVersion:
  buildDate: "2023-01-24T19:19:02Z"
  compiler: gc
  gitCommit: 9f22d4ae876173884749c0701f01340879ab3f95
  gitTreeState: clean
  gitVersion: v1.25.6-eks-48e63af
  goVersion: go1.19.5
  major: "1"
  minor: 25+
  platform: linux/amd64

Expected Behavior

should spin up a micro, small, mediam or large instance in the t3 family

Actual Behavior

spins up some flavor of c6a

Steps to Reproduce the Problem

I'm running a relatively fresh cluster and trying to install karpenter as the scaling engine. I have 1 default eks managed nodegroup for static resources like the karpenter controllers. Im trying to spin up a default karpenter provisioner in private subnets for workloads. I'm provisioning all of this in terraform and applying manifests using the kubectl provider v1.14. The manifests being applied are below. When I scale up the inflator demonstration deployment, karpenter does provision the node but its always a c6a. Subnets and security groups are being respected.

Provisioner

    apiVersion: karpenter.sh/v1alpha5
    kind: Provisioner
    metadata:
      name: default-private-workers
    spec:
      requirements::
        - key: karpenter.k8s.aws/instance-family
          operator: In
          values:
            - t3
        - key: karpenter.k8s.aws/instance-size
          operator: In
          values:
            - nano
            - micro
            - small
            - large
        - key: "kubernetes.io/arch"
          operator: In
          values:
            - amd64
        - key: karpenter.sh/capacity-type
          operator: In
          values:
            - "on-demand"
      limits:
        resources:
          cpu: 100
      providerRef:
        name: default-private-workers
      ttlSecondsAfterEmpty: 180
      ttlSecondsUntilExpired: 604800
      weight: 10

Node template

apiVersion: karpenter.k8s.aws/v1alpha1
    kind: AWSNodeTemplate
    metadata:
      name: default-private-workers
    spec:
      subnetSelector:
        karpenter.sh/discovery: ${local.eks_cluster_config.cluster_name}
        Public: "false"
      securityGroupSelector:
        aws-ids: "sg-1, sg-2"
      tags:
        karpenter.sh/discovery: ${local.eks_cluster_config.cluster_name}

Resource Specs and Logs

Karpenter logs

2023-03-06T22:49:13.130Z        INFO    controller.provisioner  found provisionable pod(s)      {"commit": "982c35f", "pods": 8}
2023-03-06T22:49:13.130Z        INFO    controller.provisioner  computed new node(s) to fit pod(s)      {"commit": "982c35f", "nodes": 1, "pods": 8}
2023-03-06T22:49:13.130Z        INFO    controller.provisioner  launching machine with 8 pods requesting {"cpu":"8155m","memory":"120Mi","pods":"11"} from types r5.24xlarge, m5a.4xlarge, c6a.24xlarge, r6idn.16xlarge, c6i.8xlarge and 181 other(s)       {"commit": "982c35f", "provisioner": "default-private-worker"}
2023-03-06T22:49:13.160Z        DEBUG   controller.provisioner.cloudprovider    discovered kubernetes version   {"commit": "982c35f", "provisioner": "default-private-worker", "kubernetes-version": "1.25"}
2023-03-06T22:49:13.192Z        DEBUG   controller.provisioner.cloudprovider    discovered new ami      {"commit": "982c35f", "provisioner": "default-private-worker", "ami": "ami-02214cc392fb65503", "query": "/aws/service/eks/optimized-ami/1.25/amazon-linux-2/recommended/image_id"}
2023-03-06T22:49:15.517Z        INFO    controller.provisioner.cloudprovider    launched new instance   {"commit": "982c35f", "provisioner": "default-private-worker", "id": "i-05c58e5e5b7370cd4", "hostname": "ip-10-20-41-151.ec2.internal", "instance-type": "c6a.4xlarge", "zone": "us-east-1a", "capacity-type": "on-demand"}
2023-03-06T22:53:06.426Z        INFO    controller.node added TTL to empty node {"commit": "982c35f", "node": "ip-10-20-41-151.ec2.internal"}

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

jonathan-innis commented 1 year ago

Is your provisioner exactly as written above? Since it looks like you have invalid YAML with two colons on L5 in the provisioner you have copied. The other thing is that the name that you have pasted from the Provisioner default-private-workers is different from the one the provisioning loop in Karpenter is launching with default-private-worker.

Can you grab the default-private-worker provisioner and see what your requirements are there?

byndcivilization commented 1 year ago

oof yup. that extra colon was the issue. the name mismatch was a fat finger when trying to pull out terraform templating into something readable. im kind of shocked the tf kubectl provider let me apply that.

aws / karpenter-provider-aws