Open wdonne opened 1 year ago
This is the current expected behavior. Karpenter provisions enough capacity for the pods and daemonsets that exist at the time of scheduling. When you add a new daemonset, for this to work properly, Karpenter would need to replace any existing nodes that the daemonset won't fit on.
We typically recommend that you set a high priority for daemonsets to cover this use case. When scaling up a daemonset, it will trigger eviction for existing pods, which will feed back into karpenter's provisioning algorithm.
I'll update the FAQ to cover this.
I forgot to mention that I tried setting the priorityClassField of the DaemonSet to system-node-critical and also once to system-cluster-critical. In both cases all pods were scheduled, but both Karpenter controllers were evicted. I will try to avoid this by changing the pod disruption budget in the values file of Karpenter’s Helm chart.
I could avoid the eviction of the Karpenter controllers, which have priority system-cluster-critical
, by giving the Prometheus DaemonSet a PriorityClass with a priority half of system-cluster-critical
. So, it works. Thanks!
This trick doesn't always work. I removed Prometheus and installed the AWS CloudWatch agent as a DaemonSet. They also have priority of 1000000000. One of the four pods can't be scheduled, but no node is added.
Here are some files that reflect the new situation. files.zip
According to the generated nodeAffinity
the node onto which the pod is supposed to be scheduled doesn't have enough memory. Shouldn't this evict other pods? There are several with priority 0. Without eviction Karpenter has no reason to provision a new node.
This feature is very necessary, Karpenter should auto adjust upon introducing a new daemonset. We should NOT have to set priority classes on every single resource in the cluster. The correct solution here would be that if a node cant fit a newly installed daemonset pod due to CPU/RAM a new bigger node should be auto ordered that is bigger and could all pods that were housed on old and nodes as well as the daemonset.
@tzneal Added https://github.com/kubernetes/website/pull/40851
I have been using the following Kyverno policy to make sure DaemonSet
s have the right priority class:
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: add-priority-class
annotations:
policies.kyverno.io/title: Add priority class for DaemonSets to help Karpenter.
policies.kyverno.io/subject: Pod
policies.kyverno.io/minversion: 1.6.0
policies.kyverno.io/description: Add priority class for DaemonSets to help Karpenter.
spec:
rules:
- name: add-priority-class-context
match:
any:
- resources:
kinds:
- DaemonSet
mutate:
patchStrategicMerge:
spec:
template:
spec:
priorityClassName: system-node-critical
Nice @wdonne , Kyverno might be interested in taking that upstream. I think it would be useful for any autoscaler.
See https://kyverno.io/policies/?policytypes=Karpenter and https://github.com/kyverno/policies/tree/main/karpenter
Hi @tzneal , thanks for the tip. I have created a pull request: https://github.com/kyverno/policies/pull/631.
If it gets merged I will create another one called "set-karpenter-non-cpu-limits". It relates to a best practice when using consolidation mode.
I have a third one that sets the annotation kubernetes.io/arch: arm64
if there is no such annotation. This way arm64
becomes the default. You would then only have to set the annotation for images that are not multi-architecture. This is not really a best practice. Would it fit in the same folder in the Kyverno policies project?
I have been using the following Kyverno policy to make sure
DaemonSet
s have the right priority class:apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: add-priority-class annotations: policies.kyverno.io/title: Add priority class for DaemonSets to help Karpenter. policies.kyverno.io/subject: Pod policies.kyverno.io/minversion: 1.6.0 policies.kyverno.io/description: Add priority class for DaemonSets to help Karpenter. spec: rules: - name: add-priority-class-context match: any: - resources: kinds: - DaemonSet mutate: patchStrategicMerge: spec: template: spec: priorityClassName: system-node-critical
this is a NOT a solution. It will not solve 99% of cases. It simply prioritizes daemonsets over non system critical items. But what if a node ordered by Karpenter cant even fit all system critical pods.... then issue not solved at all. So again , the priority class is "cute". But it is NOT a solution.
Karpenter team, I at least by all means consider your product incomplete as is. please pay attention to this comment: https://github.com/aws/karpenter-core/issues/731 This is not a feature request. This is a bug.
Community please upvote so AWS understands that it charges money for an incomplete product. Let's not give them the idea that this ticket is somehow optional.
I have been using the following Kyverno policy to make sure
DaemonSet
s have the right priority class:apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: add-priority-class annotations: policies.kyverno.io/title: Add priority class for DaemonSets to help Karpenter. policies.kyverno.io/subject: Pod policies.kyverno.io/minversion: 1.6.0 policies.kyverno.io/description: Add priority class for DaemonSets to help Karpenter. spec: rules: - name: add-priority-class-context match: any: - resources: kinds: - DaemonSet mutate: patchStrategicMerge: spec: template: spec: priorityClassName: system-node-critical
this is a NOT a solution. It will not solve 99% of cases. It simply prioritizes daemonsets over non system critical items. But what if a node ordered by Karpenter cant even fit all system critical pods.... then issue not solved at all. So again , the priority class is "cute". But it is NOT a solution.
Karpenter team, I at least by all means consider your product incomplete as is. please pay attention to this comment: #731 This is not a feature request. This is a bug.
Community please upvote so AWS understands that it charges money for an incomplete product. Let's not give them the idea that this ticket is somehow optional.
couldn't've said it better myself. we're facing the same issue too.
can't believe this bug is open for nearly a year :(
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
/remove-lifecycle stale
can't believe this bug is open for nearly a year :(
As this is an open source project: code contributions are welcome. If nobody writes the code, it doesn't get merged.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
/remove-lifecycle stale
/remove-lifecycle stale
any update?
Version
Karpenter Version: v0.22.1
Kubernetes Version: v1.24.8
Hi,
I have set up Karpenter with the following cluster configuration:
This is the provisioner:
Karpenter has currently provisioned three spot instances. When installing Prometheus with Helm chart version 19.3.1, two of the five node exporters can't be scheduled. The message is: "0/5 nodes are available: 1 Too many pods. preemption: 0/5 nodes are available: 5 No preemption victims found for incoming pod.". The Karpenter controllers didn't output any log entries.
This is the values file for the chart:
This is the live manifest of the DaemonSet of the Prometheus node exporter:
This is the live manifest of one of the pods that can't be scheduled:
I also did a test with the node selector "karpenter.sh/capacity-type: on-demand". Then one of the spot instances is deleted, but no new instance is created. The DaemonSet also doesn't create any pods.
PR aws/karpenter#1155 should have fixed the issue of DaemonSets not being part of the scaling decision, but perhaps this is a special case? The node exporter wants a pod on each node because it wants to tap telemetry.
Best regards,
Werner.
Expected Behavior
An extra node to be provisioned.
Actual Behavior
No extra ode is provisioned while two DaemonSet pods can't be scheduled.
Steps to Reproduce the Problem
I did this when there were already three Karpenter nodes, but I think you can just install Prometheus because the nodes are not full.
Resource Specs and Logs
karpenter-6d57cdbbd6-dqgcj.log karpenter-6d57cdbbd6-lsv9f.log
Community Note