I am receiving NoExecuteTaintManager events that are deleting my pod but I can't figure out why. The node is healthy and the Pod has the appropriate tolerations.
This is actually causing infinite scale up because my Pod is setup so that it uses 3 out of 4 CPUs and has a Toleration Grace Period > 0. This is to enforce a 1:1 Node to Pod ratio and causes a new node to spin up after a Pod terminates. All this is intentional for the workload we are processing. Cluster Autoscaler tries to keep the replicas == 2.
How do I figure out which taint is causing it specifically? Any then why does it think that node had that taint?
I've played around with tolerationSeconds to determine that yes this is taint related. I have a single taint specialized that I use for specific workloads. Currently the pods are being killed at exactly 600 seconds (which I have changed tolerationSeconds to be for node.kubernetes.io/unreachable and node.kubernetes.io/not-ready) however the node does not appear to undergo either of those situations.
AZRebalance has been disabled
"my-pod" is just a python app that sleeps for 10k seconds for testing purposes
I am receiving NoExecuteTaintManager events that are deleting my pod but I can't figure out why. The node is healthy and the Pod has the appropriate tolerations.
This is actually causing infinite scale up because my Pod is setup so that it uses 3 out of 4 CPUs and has a Toleration Grace Period > 0. This is to enforce a 1:1 Node to Pod ratio and causes a new node to spin up after a Pod terminates. All this is intentional for the workload we are processing. Cluster Autoscaler tries to keep the replicas == 2.
How do I figure out which taint is causing it specifically? Any then why does it think that node had that taint?
I've played around with
tolerationSeconds
to determine that yes this is taint related. I have a single taintspecialized
that I use for specific workloads. Currently the pods are being killed at exactly 600 seconds (which I have changedtolerationSeconds
to be fornode.kubernetes.io/unreachable
andnode.kubernetes.io/not-ready
) however the node does not appear to undergo either of those situations.Also posted: https://stackoverflow.com/questions/58560739/noexecutetaintmanager-falsely-deleting-pod?noredirect=1#comment103499369_58560739