search

karmada-io / karmada

Open, Multi-Cloud, Multi-Cluster Kubernetes Orchestration
https://karmada.io
Apache License 2.0
4.51k stars 892 forks source link

[Feature] Support suspend ResourceBinding when create #4937

Open Vacant2333 opened 6 months ago

Vacant2333 commented 6 months ago

What would you like to be added: I hope to add the ability to suspend ResourceBinding. If set to True, karmada-scheduler will ignore this ResourceBinding until suspend=false before starting to schedule it.

Why is this needed:

Question And Answers: Q: How does [Kueue](https://kubernetes.io/blog/2022/10/04/introducing-kueue/) provide queue capabilities to kube-scheduler? A: Taking Job as an example, Job contains a suspend field. If this field is set to True, kube-scheduler will ignore the Job until it becomes False. Kueue on the other hand, provides an additional controller. It includes queue capabilities along with sorting algorithms. Kueue considers multiple conditions (such as task priority, resource availability) before updating suspend to false and handing it over to kube-scheduler for normal scheduling thereafter.

Q: Does PropagationPolicy also need to have a suspend field? A: Yes, as I understand it, PropagationPolicy is user-facing resource. Users only need to set suspend here, and it will automatically propagate to associated resources (ResourceBinding, Work).

Q: Whats the difference with issue Ability to suspend work #4688 ? A: The focus of this issue is not to react to new changes. My focus is on pausing the distribution of the entire PropagationPolicy resource. Another point is that the Work resource is the result of Karmada scheduling. We need to ensure that the desired queue capability guarantees that resources have not yet reached the scheduling stage (considering resource quota usage).

Additional Considerations: We actually don't need to add the suspend field to PropagationPolicy; we only need to add the suspend field to ResourceBinding. Since our requirement is to perform certain actions before karmada-scheduler schedules, we just need to automatically set suspend to true when ResourceBinding is created (for example, through a webhook). This way, users don't need to set suspend when creating PropagationPolicy, which also avoids the issue of users attempting to modify suspend field after scheduling. In fact, users don't need to be aware of the suspend field; it is only provided for internal use by controllers.

Of course, this point is still open for discussion.

Additional: This feature allows for more flexible load distribution and the ability to have multiple queues. Users can have some custom controls outside of Karmada, such as capacity management for tenants (queues) and support for multi-tenancy (multiple queues).

For example, we have three queues, Q1, Q2, and Q3, with priorities of 100, 80, and 20 respectively. Under multiple queues, when several jobs are submitted, to ensure fairness among tenants, we will cyclically select the highest priority job from each queue in priority order. For instance, we first select the highest priority job from Q1, then from Q2 and Q3, and set the corresponding job's suspend flag to false, handing it over to the Karmada scheduler for scheduling.

Optional Controller:

Karmada can also provide an optional controller to implement relatively simple queue and capacity management functionality. Additionally, the sorting of queues can provide an extension point for custom plugins.

This optional controller mainly possesses the following capabilities: maintaining the priority of queues and jobs, managing the capacity of queues, modifying suspend to false after jobs are schedulable, and having a webhook that defaults to setting suspend to true when ResourceBinding/PropagationPolicy is created. In other words, once this controller is used, it automatically takes over and blocks all downstream operations.

If Karmada is willing to support this feature, I can provide an additional controller to Karmada in my free time.

whitewindmills commented 6 months ago

a little confused, I understand this is similar to the pod being scheduling gated. but different from pods and jobs, if Karmada scheduling is not required, then you only need to not create PropagationPolicy. why do you need to introduce this feature?

Monokaix commented 6 months ago

a little confused, I understand this is similar to the pod being scheduling gated. but different from pods and jobs, if Karmada scheduling is not required, then you only need to not create PropagationPolicy. why do you need to introduce this feature?

It's a different case with PropagationPolicy, PropagationPolicy is associated with scheduling info like affinity, or say that workload and PropagationPolicy are bound together, only after the PropagationPolicy created can karmada konw how to schedule them, so what the issue cares is the moment the scheduling policy has been made, but should be admitted before karmada really schedule them.

whitewindmills commented 6 months ago

@Monokaix thanks, but that's not what I want.

In k8s, Pod Scheduling Readiness is introduced mainly because some key information must be obtained after the pod is created.

For example

Users want to select the corresponding nodes based on the image architecture (arm, x86, etc.) used by the pods. so it can be achieved by following the steps below.

  1. add a scheduling gate
  2. parse the architecture supported by the image
  3. add it to the nodeAffinity
  4. remove the scheduling gate.

Now back to this use case, is there a similar situation here?

Monokaix commented 6 months ago

@Monokaix thanks, but that's not what I want.

In k8s, Pod Scheduling Readiness is introduced mainly because some key information must be obtained after the pod is created.

For example

Users want to select the corresponding nodes based on the image architecture (arm, x86, etc.) used by the pods. so it can be achieved by following the steps below.

  1. add a scheduling gate
  2. parse the architecture supported by the image
  3. add it to the nodeAffinity
  4. remove the scheduling gate.

Now back to this use case, is there a similar situation here?

With this feature, we can do more things to suspend rb scheduling, like queue capacity management, and your use case can also be resolved：）

a7i commented 6 months ago

A: The focus of this issue is not to react to new changes. My focus is on pausing the distribution of the entire PropagationPolicy resource. Another point is that the Work resource is the result of Karmada scheduling. We need to ensure that the desired queue capability guarantees that resources have not yet reached the scheduling stage (considering resource quota usage).

@Monokaix The implementation allows suspend on PropagationPolicy and ResourceBinding: https://github.com/karmada-io/karmada/pull/4838

Monokaix commented 6 months ago

A: The focus of this issue is not to react to new changes. My focus is on pausing the distribution of the entire PropagationPolicy resource. Another point is that the Work resource is the result of Karmada scheduling. We need to ensure that the desired queue capability guarantees that resources have not yet reached the scheduling stage (considering resource quota usage).

@Monokaix The implementation allows suspend on PropagationPolicy and ResourceBinding: #4838

Seems great! so what's the effect of suspending Propagation or ResourceBinding? We want to suspend in scheduling stage, is this conflict with your case?

Vacant2333 commented 2 months ago

@whitewindmills hi, looks like we can close this issue cause the proposal merged now!

RainbowMango commented 2 months ago

Yeah, I think so. Please let us know if the feature #5118 does not fit your case. /close

karmada-bot commented 2 months ago

@RainbowMango: Closing this issue.

In response to [this](https://github.com/karmada-io/karmada/issues/4937#issuecomment-2345152397): >Yeah, I think so. Please let us know if the feature #5118 does not fit your case. >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
Monokaix commented 2 months ago

/reopen The rb suspend to schedule is not done yet.

karmada-bot commented 2 months ago

@Monokaix: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to [this](https://github.com/karmada-io/karmada/issues/4937#issuecomment-2348154502): >/reopen >The rb suspend to schedule is not done yet. Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
liangyuanpeng commented 2 months ago

The rb suspend to schedule is not done yet.

/reopen

karmada-bot commented 2 months ago

@liangyuanpeng: Reopened this issue.

In response to [this](https://github.com/karmada-io/karmada/issues/4937#issuecomment-2348155786): >> The rb suspend to schedule is not done yet. > >/reopen Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
Vacant2333 commented 2 months ago

@RainbowMango hi, looks like the proposal cant suspend schedule now, we need suspend schedule ResourceBinding, it can suspend dispatch now.

RainbowMango commented 2 months ago

Is the #5218 tracking this?

Vacant2333 commented 2 months ago

Is the #5218 tracking this?

no, its another one, should i create a new issue for tracking this?