Closed asm582 closed 1 month ago
I'm assuming you are asking for Kueue to check the resources in the existing nodes. We don't plan to do this. Kueue's responsibility is quota.
That said, we have a mechanism to extend kueue, called admission checks. We have one admission check that cluster-autoscaler implements. https://kueue.sigs.k8s.io/docs/admission-check-controllers/provisioning/
Then cluster-autoscaler is responsible for checking the nodes or doing a scale up.
Btw, in general, quota management is already a good approximation for what is available in a cluster and people are using Kueue in non-autoscaled environments in production just based on that.
Since you are coming from DRA, I think we can make this feature request about DRA. Then, what we would need is a quota model for DRA resources. Not sure if this makes sense at all. But it's definitely not possible without a numeric model.
Btw, in general, quota management is already a good approximation for what is available in a cluster and people are using Kueue in non-autoscaled environments in production just based on that.
ok, so the assumption is that quotas are resources in the cluster
In a fixed sized cluster, yes. In an autoscaled cluster, it's the maximum size that the cluster can reach.
Since you are coming from DRA, I think we can make this feature request about DRA. Then, what we would need is a quota model for DRA resources. Not sure if this makes sense at all. But it's definitely not possible without a numeric model.
With DRA I think quota and physical resources may not be 1:1 and in such a scenario a quota check may pass, for instance user has a quota to acquire a slice of GPU but on the physical hardware such a slice is not realized causing a false positive dispatch if that makes sense.
I am not sure if the numeric model provides a gaurantee that a user requested slice would be made available on the node.
FYI: @asm582
I have a similar situation in the required multiple GPUs per pods. We can imagine the following situation:
I'm preparing the new requeueing strategy KEP to prevent infinity re-dispatching the above jobs.
https://github.com/kubernetes-sigs/kueue/pull/1311#issuecomment-1870531528
FYI: @asm582
I have a similar situation in the required multiple GPUs per pods. We can imagine the following situation:
- The job has 2 parallelisms and requires the sum of 10 GPUs, which means every pod requires 5 GPUs.
- Every node has only 3 GPUs although the cluster has 10 GPUs.
I'm preparing the new requeueing strategy KEP to prevent infinity re-dispatching the above jobs.
Thanks, I think this still is a case of false positive dispatch because of a lack of resource checks.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle rotten
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten
@asm582 is still fair to say that the request is for Kueue to support DRA?
OTOH, some of these "checks" can be achieved via ProvisioningRequest, once cluster-autoscaler supports DRA. Or via pod group in kube-scheduler, if the proposal progresses https://github.com/kubernetes/enhancements/pull/3371
/remove-lifecycle rotten
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle rotten
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/reopen
/remove-lifecycle rotten
Please send feedback to sig-contributor-experience at kubernetes/community.
/close not-planned
@k8s-triage-robot: Closing this issue, marking it as "Not Planned".
What would you like to be added:
When quotas are not resources it could be the case that the user has enough quota but the cluster does not have enough physical hardware resources to launch the job. This could cause a premature dispatch and lead to the creation of pending pods. In the worst case, this could lead to resource hogging in the cluster.
Why is this needed: This feature is needed to ensure that we have guaranteed workload execution post-workload dispatch.
Completion requirements:
This requirement would need modification on Kueue core dispatch logic.
This enhancement requires the following artifacts:
The artifacts should be linked in subsequent comments.