Closed anemptyair closed 1 day ago
If those taints names are known in advance, there's a --ignore-taint
flag to that effect.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/lifecycle rotten
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/lifecycle rotten
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/lifecycle rotten
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/lifecycle rotten
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/lifecycle rotten
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
/remove-lifecycle stale
@pierluigilenoci would these be the right steps to reproduce this issue?
cluster-autoscaler will not scale up thinking it can fit the workload on the tainted node.
@vadasambar exactly.
cluster-autoscaler might not be considering the newly added taints like you say. Might need deeper investigation.
I can reproduce the problem.
kubectl scale deploy node-scale-up-with-pod-anti-affinity --replicas=4
deployment.apps/node-scale-up-with-pod-anti-affinity scaled
![image](https://user-images.githubusercontent.com/34534103/226533719-38dec2da-a441-4288-b34a-42c4d143a3b7.png)
3. Taint the new node
kubectl taint no gke-cluster-1-default-pool-7c54e36a-h2gg test=true:NoSchedule node/gke-cluster-1-default-pool-7c54e36a-h2gg tainted
4. Create workload to schedule on the tainted node
kubectl create deploy nginx --image=nginx --replicas=1 deployment.apps/nginx created
Use `nodeSelector` to target the tainted node
![image](https://user-images.githubusercontent.com/34534103/226533880-bbe0e867-cb31-4cb6-bf62-c5db7e08bb75.png)
5. Pod gets stuck in `Pending` forever
![image](https://user-images.githubusercontent.com/34534103/226533079-96bcb5dd-1a21-4623-aec9-efe06b999931.png)
Example scale up workload
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: node-scale-up-with-pod-anti-affinity
namespace: default
spec:
selector:
matchLabels:
app: node-scale-up-with-pod-anti-affinity
replicas: 1
template:
metadata:
labels:
app: node-scale-up-with-pod-anti-affinity
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- node-scale-up-with-pod-anti-affinity
topologyKey: "kubernetes.io/hostname"
containers:
- name: node-scale-up-with-pod-anti-affinity
image: registry.k8s.io/pause:2.0
@vadasambar, your example is not quite 100% accurate. The new pod should be scheduled on the node with the taint because the others don't have enough resources. So you're forcing things too much. Ultimately you can cordon all nodes except the tainted one to see what happens. Or even better, put pods on the other nodes that consume enough resources so the new pod doesn't have enough room.
@pierluigilenoci maybe you can share the cluster-autoscaler log snippet around this problem? (anything that looks fishy)
@vadasambar, I had this problem about two years ago, and in the meantime, I've found external workarounds to fix it. So I no longer have a log attached to these events.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
/remove-lifecycle stale
@pierluigilenoci if this important, please bring it up in the sig-autoscaling meeting. Sorry I don't have bandwidth to look deeper into this :pray:
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
It's still a problem, but I haven't had a chance to do anything about it.
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle rotten
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten
😢
/remove-lifecycle rotten
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
ðŸ˜
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle rotten
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten
😿
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/reopen
/remove-lifecycle rotten
Please send feedback to sig-contributor-experience at kubernetes/community.
/close not-planned
@k8s-triage-robot: Closing this issue, marking it as "Not Planned".
/remove-lifecycle rotten
/reopen
@pierluigilenoci: You can't reopen an issue/PR unless you authored it or you are a collaborator.
Which component are you using?: k8s.gcr.io/autoscaling/cluster-autoscaler:v1.16.6
Is your feature request designed to solve a problem? If so describe the problem this feature should solve.: Cluster auto scale ignore node taints
We have added some taints for cluster nodes. Auto scale will failed when node are enough but with taints that pods can not be schedule into them. Can cluster auto scaler ignore nodes taints ?
Describe the solution you'd like.:
Describe any alternative solutions you've considered.:
Additional context.: