Closed FernandoMiguel closed 2 weeks ago
Applying this on my own cluster, it looks like the higher-weighted provisioner is too restrictive on instance types. If I add the following requirement to the first provisioner, I'm not even able to provision using the first provisioner
requirements:
- key: topology.kubernetes.io/zone
operator: NotIn
values:
- us-east-1a
- us-east-1d
implying that the instance types that are available using the higher-weighted provisioner are only in these AZs. What's happening with two provisioners is that topology spread is causing the scheduler to add a requirement within the provisioner's topology domains causing the launched node to be in a specific AZ (like us-east-1b
) since the first provisioner has no instance types that satisfy this requirement, it has to move to the lower-weighted provisioner.
I would suggest working to open up the higher-weighted provisioner a bit if you are trying to avoid having it fallback to the second provisioner too often.
From the perspective of the provisioner weighting mechanism, this is performing how I would expect. It's choosing the highest weighted provisioner in zones where it can and not in zones where it can't.
the 1st one has still a very large number of instance types. and all the instances spun up from the 2nd could fit in the 1st. sure topology is at play here, but both continue to spin up nodes on all 5 AZs.
and the pod has ScheduleAnyway
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: ScheduleAnyway
so I remain unsure what difference did it make picking up the 2nd provisioner, @jonathan-innis
and the pod has ScheduleAnyway
We still try to satisfy the topology constraint across all provisioners before we relax the constraint, even if it's a preferred constraint; so, the fact that it's a ScheduleAnyway
constraint doesn't make a difference in terms of falling back between provisioners.
the 1st one has still a very large number of instance types, and all the instances spun up from the 2nd could fit in the 1st.
That's a good point. It looks like I may be working under constrained capacity pools right now. Will take another look since you're right, both provisioners launch with the same instance type in the same AZ in your example.
My other guess is that we are hitting provisioner limits and falling back to the second provisioner. During scheduling, we assume worst-case among the proposed instance types when subtracting from the limits so that we don't overshoot the limits after launch.
If you bump up the cpu
limit or remove it entirely do you see the same fallback behavior?
i'll re-try with 10k CPUs... but none of the provisioners ever hit 100 CPUs, according to the info in each of them. even the total of both is less than 100
During scheduling, we assume worst-case among the proposed instance types when subtracting from the limits so that we don't overshoot the limits after launch.
but none of the provisioners ever hit 100 CPUs
This means that even if we don't hit the limits after launch, we may still consider a provisioner to have hit its limits prior to launch since we take the largest instance possible to be the representative until after we actually know what we launched.
interesting to learn
@FernandoMiguel Did removing the provisioner limits fix the issue?
@jonathan-innis havent been able to test in detail, but havent seen this happen with 10k limit set, instead of 100
We're picking up a fix for this issue so I'm going to re-open it for tracking.
I think the proposed solution here was that we can just exit scheduling early when we have simulated capacity and we recognize that one of our provisioners might exceed the limit.
This issue has been inactive for 14 days. StaleBot will close this stale issue after 14 more days of inactivity.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle rotten
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/reopen
/remove-lifecycle rotten
Please send feedback to sig-contributor-experience at kubernetes/community.
/close not-planned
@k8s-triage-robot: Closing this issue, marking it as "Not Planned".
Version
Karpenter Version: v0.20.0 Kubernetes Version: v1.24.0 Server Version: version.Info{Major:"1", Minor:"24+", GitVersion:"v1.24.7-eks-fb459a0", GitCommit:"c240013134c03a740781ffa1436ba2688b50b494", GitTreeState:"clean", BuildDate:"2022-10-24T20:36:26Z", GoVersion:"go1.18.7", Compiler:"gc", Platform:"linux/amd64"}
Expected Behavior
I have two providers , one with a smaller number of instances, and one much bigger the 1st one has a weight of 50
i've just spun up 20 pods/nodes and noticed it was using both providers and picking instances that could fit in the 1st one. trying to understand why that happened.
should the weight be bigger?
Actual Behavior
N/A
Steps to Reproduce the Problem
N/A
Resource Specs and Logs
p1
p2
pod
Community Note