instance size preference - why it chooses the biggest ec2 when only 1 pod is requested

[funes79]

Version

Karpenter Version: v0.18.1

Kubernetes Version: v1.0.0

Expected Behavior

I would expect that when 1 pod is requested a "big enough" EC2 will be launched, but not the biggest one. What I see in the logs that every time a small spark job starts (executor.instances=1) karpenter launches a 48CPU machine. Isnt that an overkill?

Actual Behavior

Karpenter decideds to run r6id.12xlarge all the time.

INFO controller.provisioning Found 1 provisionable pod(s) {"commit": "51becf8-dirty"}
--
Wed, Nov 30 2022 2:19:32 pm | 2022-11-30T13:19:32.399Z INFO controller.provisioning Computed 1 new node(s) will fit 1 pod(s) {"commit": "51becf8-dirty"}
Wed, Nov 30 2022 2:19:32 pm | 2022-11-30T13:19:32.399Z INFO controller.provisioning Launching node with 1 pods requesting {"cpu":"2125m","memory":"2Gi","pods":"4"} from types r6id.2xlarge, m6id.2xlarge, r6id.4xlarge, m6id.4xlarge, m6id.8xlarge and 5 other(s) {"commit": "51becf8-dirty", "provisioner": "spark-prov"}
Wed, Nov 30 2022 2:19:34 pm | 2022-11-30T13:19:34.767Z INFO controller.provisioning.cloudprovider Launched instance: i-045a4cefaaac678be, hostname: ip-11-5-150-196.eu-west-1.compute.internal, type: r6id.12xlarge, zone: eu-west-1b, capacityType: spot {"commit": "51becf8-dirty", "provisioner": "spark-prov"}
Wed, Nov 30 2022 2:21:26 pm | 2022-11-30T13:21:26.932Z INFO controller.node Added TTL to empty node {"commit": "51becf8-dirty", "node": "ip-11-5-150-196.eu-west-1.compute.internal"}
Wed, Nov 30 2022 2:21:26 pm | 2022-11-30T13:21:26.955Z INFO controller.node Added TTL to empty node {"commit": "51becf8-dirty", "node": "ip-11-5-150-196.eu-west-1.compute.internal"}

Steps to Reproduce the Problem

The provisioner we use in this case:

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: spark-prov
spec:
  providerRef:                                
    name: spark-template-nvme
  # Application requirements
  requirements:
    - key: karpenter.sh/capacity-type
      operator: In
      values:
        - spot
    - key: kubernetes.io/arch
      operator: In
      values:
        - amd64  
    - key: "node.kubernetes.io/instance-type" 
      operator: In 
      values: ["m6id.large", "m6id.2xlarge", "m6id.4xlarge", "m6id.8xlarge", "m6id.12xlarge", "m6id.16xlarge","r6id.2xlarge", "r6id.4xlarge", "r6id.8xlarge", "r6id.12xlarge", "r6id.16xlarge"]
  limits:
    resources:
      cpu: '600'
  taints:
    - key: spark
      effect: NoSchedule  
  labels:
    node-group: spark-prod
  ttlSecondsAfterEmpty: 60

Resource Specs and Logs

See the log output in actual behaviour

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[FernandoMiguel]

spot market can be weird. but even at spot prices, seems overkill, indeed.

[ellistarn]

In karpenter 1.19, we use a new fleet strategy that optimizes more heavily for price, rather than spot pool depth. Read more here: https://github.com/aws/karpenter/pull/2835