search

kubernetes-sigs / kueue

Kubernetes-native Job Queueing
https://kueue.sigs.k8s.io
Apache License 2.0
1.46k stars 262 forks source link

Add resource check before dispatching workloads #1538

Closed asm582 closed 1 month ago

asm582 commented 10 months ago

What would you like to be added:

When quotas are not resources it could be the case that the user has enough quota but the cluster does not have enough physical hardware resources to launch the job. This could cause a premature dispatch and lead to the creation of pending pods. In the worst case, this could lead to resource hogging in the cluster.

Why is this needed: This feature is needed to ensure that we have guaranteed workload execution post-workload dispatch.

Completion requirements:

This requirement would need modification on Kueue core dispatch logic.

This enhancement requires the following artifacts:

The artifacts should be linked in subsequent comments.

alculquicondor commented 10 months ago

I'm assuming you are asking for Kueue to check the resources in the existing nodes. We don't plan to do this. Kueue's responsibility is quota.

That said, we have a mechanism to extend kueue, called admission checks. We have one admission check that cluster-autoscaler implements. https://kueue.sigs.k8s.io/docs/admission-check-controllers/provisioning/

Then cluster-autoscaler is responsible for checking the nodes or doing a scale up.

alculquicondor commented 10 months ago

Btw, in general, quota management is already a good approximation for what is available in a cluster and people are using Kueue in non-autoscaled environments in production just based on that.

alculquicondor commented 10 months ago

Since you are coming from DRA, I think we can make this feature request about DRA. Then, what we would need is a quota model for DRA resources. Not sure if this makes sense at all. But it's definitely not possible without a numeric model.

asm582 commented 10 months ago

Btw, in general, quota management is already a good approximation for what is available in a cluster and people are using Kueue in non-autoscaled environments in production just based on that.

ok, so the assumption is that quotas are resources in the cluster

alculquicondor commented 10 months ago

In a fixed sized cluster, yes. In an autoscaled cluster, it's the maximum size that the cluster can reach.

asm582 commented 10 months ago

Since you are coming from DRA, I think we can make this feature request about DRA. Then, what we would need is a quota model for DRA resources. Not sure if this makes sense at all. But it's definitely not possible without a numeric model.

With DRA I think quota and physical resources may not be 1:1 and in such a scenario a quota check may pass, for instance user has a quota to acquire a slice of GPU but on the physical hardware such a slice is not realized causing a false positive dispatch if that makes sense.

I am not sure if the numeric model provides a gaurantee that a user requested slice would be made available on the node.

tenzen-y commented 10 months ago

FYI: @asm582

I have a similar situation in the required multiple GPUs per pods. We can imagine the following situation:

I'm preparing the new requeueing strategy KEP to prevent infinity re-dispatching the above jobs.

https://github.com/kubernetes-sigs/kueue/pull/1311#issuecomment-1870531528

asm582 commented 10 months ago

FYI: @asm582

I have a similar situation in the required multiple GPUs per pods. We can imagine the following situation:

  • The job has 2 parallelisms and requires the sum of 10 GPUs, which means every pod requires 5 GPUs.
  • Every node has only 3 GPUs although the cluster has 10 GPUs.

I'm preparing the new requeueing strategy KEP to prevent infinity re-dispatching the above jobs.

#1311 (comment)

Thanks, I think this still is a case of false positive dispatch because of a lack of resource checks.

k8s-triage-robot commented 7 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 6 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

alculquicondor commented 6 months ago

@asm582 is still fair to say that the request is for Kueue to support DRA?

OTOH, some of these "checks" can be achieved via ProvisioningRequest, once cluster-autoscaler supports DRA. Or via pod group in kube-scheduler, if the proposal progresses https://github.com/kubernetes/enhancements/pull/3371

alculquicondor commented 6 months ago

/remove-lifecycle rotten

k8s-triage-robot commented 3 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 2 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 1 month ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 1 month ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes-sigs/kueue/issues/1538#issuecomment-2381444010): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.