Jobs do not get scheduled even though there are available resources on the cluster.
Steps to Reproduce
Submit a job when the cluster is at relatively high load (>~80% of resources are utilized).
Expected Results
If there are resources that meet the job's requirements at least some of it tasks should start.
Actual Results
The job remains in queued state despite there being available resources in the cluster.
Additonal Comments
Surprisingly, this phenomenon happens both for Queued and Balanced mode. Even more surprisingly, momentarily switching the Scheduling Policy to the other option and back (regardless of the original mode) solves this issue and the queued jobs will get scheduled.
Not sure if this is still an issue. Please collect the HPC Pack version info and HpcScheduler service logs. Send to hpcpack@microsoft.com for further investigation.
Problem Description
Jobs do not get scheduled even though there are available resources on the cluster.
Steps to Reproduce
Submit a job when the cluster is at relatively high load (>~80% of resources are utilized).
Expected Results
If there are resources that meet the job's requirements at least some of it tasks should start.
Actual Results
The job remains in queued state despite there being available resources in the cluster.
Additonal Comments
Surprisingly, this phenomenon happens both for Queued and Balanced mode. Even more surprisingly, momentarily switching the Scheduling Policy to the other option and back (regardless of the original mode) solves this issue and the queued jobs will get scheduled.