Closed mattwelborn closed 1 month ago
The podgroup of higher job didn't create pods because you have enabled the enqueue
action, refer to https://github.com/volcano-sh/volcano/blob/master/docs/design/delay-pod-creation.md for more details.
enqeue
action will block pods to enqueue when cluster resources not enough, you can remove it and try it anoher time: )
/close
@Monokaix: Closing this issue.
Description
I have two types of jobs:
When I submit many low-priority jobs and then submit one high-priority job, the high-priority job does not run. It does not create pods. Its podgroup has an event which says "resource in cluster is overused".
Steps to reproduce the issue
node-1
andnode-2
.pc-low
has priority 10.pc-high
has priority 100.q-low
has weight 1.q-high
has weight 100.job-low
which usespc-low
,q-low
, can only be scheduled ontonode-1
, and which sleeps for 5 minutes.job-high
which usespc-high
,q-high
, can only be scheduled ontonode-2
, and which sleeps forever.job-low
.job-high
.Describe the results you received and expected
Results received: many pods are created for the copies of
job-low
until there are ~80 pending pods corresponding tojob-low
. These copies ofjob-low
run onnode-1
, whilenode-2
remains idle. The one copy ofjob-high
creates a podgroup, but the podgroup fails to create a pod. Instead, it repeatedly gives the error "resource in cluster is overused". Even as the jobs corresponding tojob-low
complete,job-high
does not run.Expected results: the copies of
job-low
should run in order of submission onnode-1
. (This happens.) The one copy ofjob-high
should immediately run onnode-2
. (This does not happen.)What version of Volcano are you using?
v1.9.0
Any other relevant information
I believe that my Volcano scheduler config is the default: