Karpenter keep scheduling pod with PVC on node that reach max EBS volume

Description

Observed Behavior: When a node reach the maximum allocatable volume, Karpenter keep trying to schedule pod with PVC on that node.

Expected Behavior: Karpenter should scale up a new node to schedule new pod with volumes.

Reproduction Steps (Please include YAML):

Deploy a new nodepool:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: test
spec:
weight: 10
template:
metadata:
  labels:
    role:test
spec:
  nodeClassRef:
    group: karpenter.k8s.aws
    kind: EC2NodeClass
    name: firstclass
  startupTaints:
    - key: ebs.csi.aws.com/agent-not-ready
      effect: NoExecute
  taints:
    - key: test
      effect: NoSchedule
  expireAfter: Never
  terminationGracePeriod: 48h
  requirements:
    - key: "karpenter.k8s.aws/instance-family"
      operator: In
      values: ["r4", "r5"]
    - key: "karpenter.sh/capacity-type"
      operator: In
      values: ["on-demand"]
    - key: karpenter.k8s.aws/instance-size
      operator: In
      values: ["xlarge", "2xlarge"]
disruption:
consolidationPolicy: WhenEmptyOrUnderutilized
consolidateAfter: Never # Added to allow additional control over consolidation aggressiveness
weight: 10

Deploy a statefulset that target the node

apiVersion: apps/v1
kind: StatefulSet
metadata:
name: test
labels:
app: my-app
namespace: kube-system
spec:
replicas: 25
selector:
matchLabels:
  app: my-app
template:
metadata:
  labels:
    app: my-app
spec:
  containers:
  - name: my-app-container
    image: busybox
    command: ["/bin/sh", "-c", "while true; do sleep1; done"]
    securityContext:
      runAsNonRoot: true
      runAsUser: 1000
      allowPrivilegeEscalation: false 
      capabilities:
        drop:
          - ALL
      seccompProfile:
        type: RuntimeDefault
    volumeMounts:
    - mountPath: /mnt/data
      name: my-pvc
  tolerations:
  - key: "test"
    operator: "Exists"
    effect: "NoSchedule"
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: role
            operator: In
            values:
            - test
volumeClaimTemplates:
- metadata:
    name: my-pvc
  spec:
    storageClassName: gp2-1a
    accessModes: [ "ReadWriteOnce" ]
    resources:
      requests:
        storage: 10Gi

The storageClass look like this:

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: gp2-1a
parameters:
type: gp2
provisioner: ebs.csi.aws.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

Versions:

Karpenter Controller Version: 1.0.6
Kubernetes Version (kubectl version): v1.30.3
Server Version: v1.30.4-eks-a737599
aws-ebs-csi-driver:v1.35.0
Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

kubernetes-sigs / karpenter

Karpenter keep scheduling pod with PVC on node that reach max EBS volume #1748

Description