Open tenzen-y opened 1 year ago
I like this idea and I don't think we will face the problems you mention. When ResourceQuotas are hit, the effect is that the pod can't even be created. The scheduler doesn't even get to see those pods. And because we will only allow jobs that satisfy the resource quota to be created, they won't hit the limit. However, the open question is what to do with flavors. For it's simplest form, probably there would only be one flavor for which all resources are defined. And I guess we would create a ClusterQueue for each ResourceFlavor. It could be an optional mode of operation.
When ResourceQuotas are hit, the effect is that the pod can't even be created. The scheduler doesn't even get to see those pods. And because we will only allow jobs that satisfy the resource quota to be created, they won't hit the limit.
Ah, I see. Thank you for clarifying.
However, the open question is what to do with flavors. For it's simplest form, probably there would only be one flavor for which all resources are defined. And I guess we would create a ClusterQueue for each ResourceFlavor.
Uhm. In that case, we can not create ClusterQueues with multiple ResourceFlavors if batch admins select the optional mode of operation, right?
Unless there are some annotations in the ResoureQuota, but that could be a later extension
Unless there are some annotations in the ResoureQuota, but that could be a later extension
Sounds good.
Unless there are some annotations in the ResoureQuota, but that could be a later extension
Could you explain this design in more detail? Sorry to bother you after such a long time.
How to balance clusterqueue quota vs. namespace ResourceQuota is a bit complicated, especially when clusterqueue is a cluster-level resource.
Would NS ResourceQuota be another admission check?
Hi, @tenzen-y
One of the import use case as below:
In a namespace, there may be different types of workloads(I call it "hybrid-workload") consuming hardware accelerators:
So below situation happens:
kueue
can management the batch jobs, to limit them under quota of the ClusterQueue's FlavorQuotas.The challenges are :
In short, I think it's valuable to address the challenges , so that user can leverage kueue
in multi-tenant circumstance with
"hybrid-workload" :-)
Would NS ResourceQuota be another admission check?
@kerthcet I think so too. Additional admission checks would resolve this issue. @alculquicondor Any concerns?
@panpan0000 I agree with issues by multi-dimensions quota management.
However, I think that your use cases might be sufficient once you can create CluserQueue against each namespaces (tenants). Then, you can construct cohorts against multiple ClusterQueue.
interactive notebook model serving deployments
Since we support naked pods, you can manage those workloads by kueue. Ideally, I would suggest implementing CutomJob controllers to manage those workloads, then you can implement the kueue workload controllers to manage those CustomJobs by kueue.
I'm not against supporting ResourceQuota by admission checks, and I agree with the ResourceQuota support. I'm sure there are use cases in which we want to manage workloads by kueue + ResourceQuota, although, in an ideal world, all workloads are managed by the kueue's flavorQuotas.
Also, we need to support elastic jobs to support model serving deployment with autoscaling semantics.
What we hope to mitigate here is make job scheduling more smoothly, rather than admitted by kueue but rejected the resourceQuota admission plugin. But the tricky thing here is resourceQuota is too flexible, it has a bunch of policies and kueue can not simply sync with. So admission check might be the simplest thing we can boot with.
So admission check might be the simplest thing we can boot with.
I agree. An additional admission check would be better. As another use case, existing resources managed by ResourceQuotas could easily adapt to the kueue.
In the adaption to the existing environments, I think that the ResourceQuota support has an advantage in terms of using all kueue features over using naked pod integration since our naked pod integration doesn't support all kueue features.
However, I'm on the fence about whether we should support ResourceQuota by AdmissionCheck since using ResourceQuota is a temporary measure since as I mentioned above, ideally, all resources are managed by Kueue's flavorQuotas.
@alculquicondor What do you think about supporting ResourceQuota by AdmissionCheck for easy adaptation to the existing environment?
I don't think we should implement it.
As explained, it will be possible for someone to implement it using AdmissionChecks, but I don't think we should support it in core Kueue. I would accept a controller in the cmd/experimental package though.
I don't think we should implement it.
As explained, it will be possible for someone to implement it using AdmissionChecks, but I don't think we should support it in core Kueue. I would accept a controller in the cmd/experimental package though.
The cmd/experimental
sounds much more reasonable. SGTM
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle rotten
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten
/remove-lifecycle rotten Still valid I think
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle rotten
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten
What would you like to be added: Batch administrators can install ResourceQuota to set constraints for total requests in the namespace.
We should consider ResourceQuotas when admitting Workloads.
Why is this needed:
For now, we may sometime face the following problems:1. if we useSequential Admission with Ready Pods
and ResourceQuotas together, we may face deadlocks.2. Many unschedulable pods (with pending status) could be created.Completion requirements:
This enhancement requires the following artifacts:
The artifacts should be linked in subsequent comments.