Closed WxFang closed 1 month ago
Update on the context: we have also deployed a daemonset called cost-placeholder. The idea is to deploy low-priority pods for better cost attribution of idle cost. And the spec looks as below:
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: nodepool
operator: In
values:
- gpu-p5-48-llm
resources:
limits:
cpu: "191"
memory: 1995G
nvidia.com/gpu: "8"
requests:
cpu: "191"
memory: 1995G
nvidia.com/gpu: "8"
priorityClassName: low-priority
restartPolicy: Always
schedulerName: default-scheduler
This hacky solution is working fine with CAS.
I'm going to close this issue out here, could you open this up as a feature request in the upstream repo, kubernetes-sigs/karpenter? This is currently intended behavior, Karpenter is going to attempt to schedule all pods, including those with a low-priority.
Description
Observed Behavior: We're trying to migrate gpu nodepool (p5.48xlarge) to karpenter. But pods kept pending and I can see scheduling events. Basically we should have 6 daemonset pods consuming very little cpu/ram only. And each workload pod requests 8 gpu which means one node can only have 7 pods in total. But in the event log, you can see workload pod is some double counted.
Expected Behavior: daemonset overhead should not include 8 gpu and pod count should be 6.
Reproduction Steps (Please include YAML):
Versions:
Chart Version: 0.37.0
Kubernetes Version (
kubectl version
): 1.29Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment