search

kubernetes-sigs / kueue

Kubernetes-native Job Queueing
https://kueue.sigs.k8s.io
Apache License 2.0
1.32k stars 230 forks source link

Consider ResourceQuota when admitting workloads #696

Open tenzen-y opened 1 year ago

tenzen-y commented 1 year ago

What would you like to be added: Batch administrators can install ResourceQuota to set constraints for total requests in the namespace.

We should consider ResourceQuotas when admitting Workloads.

Why is this needed: For now, we may sometime face the following problems:

1. if we use Sequential Admission with Ready Pods and ResourceQuotas together, we may face deadlocks. 2. Many unschedulable pods (with pending status) could be created.

Completion requirements:

This enhancement requires the following artifacts:

The artifacts should be linked in subsequent comments.

alculquicondor commented 1 year ago

I like this idea and I don't think we will face the problems you mention. When ResourceQuotas are hit, the effect is that the pod can't even be created. The scheduler doesn't even get to see those pods. And because we will only allow jobs that satisfy the resource quota to be created, they won't hit the limit. However, the open question is what to do with flavors. For it's simplest form, probably there would only be one flavor for which all resources are defined. And I guess we would create a ClusterQueue for each ResourceFlavor. It could be an optional mode of operation.

tenzen-y commented 1 year ago

When ResourceQuotas are hit, the effect is that the pod can't even be created. The scheduler doesn't even get to see those pods. And because we will only allow jobs that satisfy the resource quota to be created, they won't hit the limit.

Ah, I see. Thank you for clarifying.

tenzen-y commented 1 year ago

However, the open question is what to do with flavors. For it's simplest form, probably there would only be one flavor for which all resources are defined. And I guess we would create a ClusterQueue for each ResourceFlavor.

Uhm. In that case, we can not create ClusterQueues with multiple ResourceFlavors if batch admins select the optional mode of operation, right?

alculquicondor commented 1 year ago

Unless there are some annotations in the ResoureQuota, but that could be a later extension

tenzen-y commented 1 year ago

Unless there are some annotations in the ResoureQuota, but that could be a later extension

Sounds good.

samzong commented 9 months ago

Unless there are some annotations in the ResoureQuota, but that could be a later extension

Could you explain this design in more detail? Sorry to bother you after such a long time.

How to balance clusterqueue quota vs. namespace ResourceQuota is a bit complicated, especially when clusterqueue is a cluster-level resource.

kerthcet commented 9 months ago

Would NS ResourceQuota be another admission check?

panpan0000 commented 9 months ago

Hi, @tenzen-y

One of the import use case as below:

In a namespace, there may be different types of workloads(I call it "hybrid-workload") consuming hardware accelerators:

So below situation happens:

The challenges are :

In short, I think it's valuable to address the challenges , so that user can leverage kueue in multi-tenant circumstance with
"hybrid-workload" :-)

tenzen-y commented 9 months ago

Would NS ResourceQuota be another admission check?

@kerthcet I think so too. Additional admission checks would resolve this issue. @alculquicondor Any concerns?

tenzen-y commented 9 months ago

@panpan0000 I agree with issues by multi-dimensions quota management.

However, I think that your use cases might be sufficient once you can create CluserQueue against each namespaces (tenants). Then, you can construct cohorts against multiple ClusterQueue.

interactive notebook model serving deployments

Since we support naked pods, you can manage those workloads by kueue. Ideally, I would suggest implementing CutomJob controllers to manage those workloads, then you can implement the kueue workload controllers to manage those CustomJobs by kueue.

tenzen-y commented 9 months ago

I'm not against supporting ResourceQuota by admission checks, and I agree with the ResourceQuota support. I'm sure there are use cases in which we want to manage workloads by kueue + ResourceQuota, although, in an ideal world, all workloads are managed by the kueue's flavorQuotas.

tenzen-y commented 9 months ago

Also, we need to support elastic jobs to support model serving deployment with autoscaling semantics.

https://github.com/kubernetes-sigs/kueue/issues/77

kerthcet commented 9 months ago

What we hope to mitigate here is make job scheduling more smoothly, rather than admitted by kueue but rejected the resourceQuota admission plugin. But the tricky thing here is resourceQuota is too flexible, it has a bunch of policies and kueue can not simply sync with. So admission check might be the simplest thing we can boot with.

tenzen-y commented 8 months ago

So admission check might be the simplest thing we can boot with.

I agree. An additional admission check would be better. As another use case, existing resources managed by ResourceQuotas could easily adapt to the kueue.

In the adaption to the existing environments, I think that the ResourceQuota support has an advantage in terms of using all kueue features over using naked pod integration since our naked pod integration doesn't support all kueue features.

However, I'm on the fence about whether we should support ResourceQuota by AdmissionCheck since using ResourceQuota is a temporary measure since as I mentioned above, ideally, all resources are managed by Kueue's flavorQuotas.

@alculquicondor What do you think about supporting ResourceQuota by AdmissionCheck for easy adaptation to the existing environment?

alculquicondor commented 8 months ago

I don't think we should implement it.

As explained, it will be possible for someone to implement it using AdmissionChecks, but I don't think we should support it in core Kueue. I would accept a controller in the cmd/experimental package though.

tenzen-y commented 8 months ago

I don't think we should implement it.

As explained, it will be possible for someone to implement it using AdmissionChecks, but I don't think we should support it in core Kueue. I would accept a controller in the cmd/experimental package though.

The cmd/experimental sounds much more reasonable. SGTM

k8s-triage-robot commented 5 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 4 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

kerthcet commented 4 months ago

/remove-lifecycle rotten Still valid I think

k8s-triage-robot commented 1 month ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 1 week ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten