Support Pod Scheduling Readiness

[ykcai-daniel]

What would you like to be added:

As mentioned in #3187, we hope to support .spec.schedulingGates flag of a pod. Scheduling Gates is a beta feature in Kubernetes v1.27. In kube-scheduler, if this field of a Pod is not empty, the pod will not be scheduled by it. Volcano should also support this behavior.

Why is this needed:

3187 use the situation of an external quota manager to shouw why schedulingGates is needed.

By specifying/removing a Pod's .spec.schedulingGates, user can control when a Pod is ready to be considered for scheduling. Here is a typical scenario from community user.

Problem Statement: We’ve implemented an external quota manager responsible for reviewing all incoming POD requests for capacity/quota requirements. Only once these requests receive approval from the quota manager are they considered eligible for scheduling. Consequently, we intend to leverage the pods schedulingGates feature to realize this functionality.

It seems that Volcano is not adhering to this feature. It proceeds to schedule pods without honoring schedulingGates flag. This behavior is in contrast to the default Kubernetes kube-scheduler, which functions as expected. Even though these pods have the schedulingGates added and were assigned to a node by the Volcano scheduler, it appears that they are subsequently rejected by the Kubernetes API server. I believe this error is occurring after the execution of the Volcano scheduler.

Failed to bind pod <namsepace/pod-name> to node : &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:"", RemainingItemCount:(int64)(nil)}, Status:"Failure", Message:"Operation cannot be fulfilled on pods/binding \”app-name\”: pod pod-name has non-empty .spec.schedulingGates", Reason:"Conflict", Details:(v1.StatusDetails)(0x140002188a0), Code:409}}

[Monokaix]

/assign @ykcai-daniel