HyperQueue submission gets stuck if workers fail

For some time, I am observing this situation. HyperQueue works fine for some time and then something happens. Jobs are coming to the HyperQueue, are buffered there but no workers are submitted even though the allocation queue has backlog set to 10. The situation from this morning:

there are no jobs in the batch

[svatosm@login3.karolina ~]$ squeue --me
         JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)

list of workers confirms that

[svatosm@login3.karolina ~]$ hq-v0.19.0-linux-x64/hq worker list
+----+-------+----------+-----------+---------+----------------+
| ID | State | Hostname | Resources | Manager | Manager Job ID |
+----+-------+----------+-----------+---------+----------------+

there is enough jobs buffered

[svatosm@login3.karolina ~]$ hq-v0.19.0-linux-x64/hq job list
+------+------+---------+-------+
| ID   | Name | State   | Tasks |
+------+------+---------+-------+
| 4565 | bash | WAITING | 1     |
| 4567 | bash | WAITING | 1     |
| 4573 | bash | WAITING | 1     |
| 4584 | bash | WAITING | 1     |
| 4596 | bash | WAITING | 1     |
| 4597 | bash | WAITING | 1     |
| 4600 | bash | WAITING | 1     |
| 4609 | bash | WAITING | 1     |
| 4618 | bash | WAITING | 1     |
| 4626 | bash | WAITING | 1     |
| 4634 | bash | WAITING | 1     |
| 4635 | bash | WAITING | 1     |
| 4636 | bash | WAITING | 1     |
| 4637 | bash | WAITING | 1     |
| 4638 | bash | WAITING | 1     |
| 4639 | bash | WAITING | 1     |
| 4640 | bash | WAITING | 1     |
| 4641 | bash | WAITING | 1     |
| 4642 | bash | WAITING | 1     |
| 4644 | bash | WAITING | 1     |
| 4645 | bash | WAITING | 1     |
| 4648 | bash | WAITING | 1     |
| 4651 | bash | WAITING | 1     |
| 4652 | bash | WAITING | 1     |
| 4653 | bash | WAITING | 1     |
| 4654 | bash | WAITING | 1     |
| 4655 | bash | WAITING | 1     |
| 4656 | bash | WAITING | 1     |
| 4657 | bash | WAITING | 1     |
| 4659 | bash | WAITING | 1     |
| 4660 | bash | WAITING | 1     |
| 4663 | bash | WAITING | 1     |
| 4664 | bash | WAITING | 1     |
| 4666 | bash | WAITING | 1     |
| 4668 | bash | WAITING | 1     |
| 4670 | bash | WAITING | 1     |
| 4674 | bash | WAITING | 1     |
| 4677 | bash | WAITING | 1     |
| 4679 | bash | WAITING | 1     |
| 4680 | bash | WAITING | 1     |
| 4682 | bash | WAITING | 1     |
| 4683 | bash | WAITING | 1     |
| 4684 | bash | WAITING | 1     |
| 4685 | bash | WAITING | 1     |
| 4688 | bash | WAITING | 1     |
| 4689 | bash | WAITING | 1     |
| 4690 | bash | WAITING | 1     |
| 4691 | bash | WAITING | 1     |
| 4692 | bash | WAITING | 1     |
| 4693 | bash | WAITING | 1     |
| 4694 | bash | WAITING | 1     |
| 4695 | bash | WAITING | 1     |
| 4696 | bash | WAITING | 1     |
| 4697 | bash | WAITING | 1     |
| 4698 | bash | WAITING | 1     |
| 4699 | bash | WAITING | 1     |
| 4700 | bash | WAITING | 1     |
| 4701 | bash | WAITING | 1     |
| 4702 | bash | WAITING | 1     |
| 4703 | bash | WAITING | 1     |
| 4704 | bash | WAITING | 1     |
| 4705 | bash | WAITING | 1     |
| 4706 | bash | WAITING | 1     |
| 4707 | bash | WAITING | 1     |
| 4708 | bash | WAITING | 1     |
| 4709 | bash | WAITING | 1     |
| 4710 | bash | WAITING | 1     |
+------+------+---------+-------+
There are 4710 jobs in total. Use `--all` to display all jobs.

the allocation queue is set with backlog of 10 and it is running

[svatosm@login3.karolina ~]$ hq-v0.19.0-linux-x64/hq alloc list
+----+---------+--------------+-------------------+------------------+-----------+---------+------+----------------------------------+
| ID | State   | Backlog size | Workers per alloc | Max worker count | Timelimit | Manager | Name | Args                             |
+----+---------+--------------+-------------------+------------------+-----------+---------+------+----------------------------------+
| 1  | RUNNING | 10           | 1                 | 0                | 12h       | SLURM   |      | -AOPEN-29-6,-pqcpu_preempt,-c128 |
+----+---------+--------------+-------------------+------------------+-----------+---------+------+----------------------------------+

the content of the debug log shows that HyperQueue cannot find workers it sent:

[2024-10-01T06:07:58.724Z DEBUG hyperqueue::server::autoalloc::process] Attempt to refresh allocations of queue 1
[2024-10-01T06:07:58.724Z DEBUG hyperqueue::server::autoalloc::queue::slurm] Running Slurm command `scontrol show job 1739716`
[2024-10-01T06:07:58.734Z DEBUG hyperqueue::server::autoalloc::queue::slurm] Running Slurm command `scontrol show job 1739715`
[2024-10-01T06:07:58.743Z DEBUG hyperqueue::server::autoalloc::queue::slurm] Running Slurm command `scontrol show job 1739718`
[2024-10-01T06:07:58.753Z DEBUG hyperqueue::server::autoalloc::queue::slurm] Running Slurm command `scontrol show job 1739717`
[2024-10-01T06:07:58.762Z DEBUG hyperqueue::server::autoalloc::queue::slurm] Running Slurm command `scontrol show job 1739711`
[2024-10-01T06:07:58.771Z DEBUG hyperqueue::server::autoalloc::queue::slurm] Running Slurm command `scontrol show job 1739702`
[2024-10-01T06:07:58.779Z DEBUG hyperqueue::server::autoalloc::queue::slurm] Running Slurm command `scontrol show job 1739704`
[2024-10-01T06:07:58.787Z DEBUG hyperqueue::server::autoalloc::queue::slurm] Running Slurm command `scontrol show job 1739710`
[2024-10-01T06:07:58.796Z DEBUG hyperqueue::server::autoalloc::queue::slurm] Running Slurm command `scontrol show job 1739700`
[2024-10-01T06:07:58.804Z DEBUG hyperqueue::server::autoalloc::queue::slurm] Running Slurm command `scontrol show job 1739712`
[2024-10-01T06:07:58.812Z DEBUG hyperqueue::server::autoalloc::process] Allocations of 1 have been refreshed: Ok(Map({"1739710": Err(scontrol execution failed

Caused by:
    Exit code: 1
    Stderr: slurm_load_jobs error: Invalid job id specified
    Stdout: ), "1739717": Err(scontrol execution failed

Caused by:
    Exit code: 1
    Stderr: slurm_load_jobs error: Invalid job id specified
    Stdout: ), "1739711": Err(scontrol execution failed

Caused by:
    Exit code: 1
    Stderr: slurm_load_jobs error: Invalid job id specified
    Stdout: ), "1739700": Err(scontrol execution failed

Caused by:
    Exit code: 1
    Stderr: slurm_load_jobs error: Invalid job id specified
    Stdout: ), "1739715": Err(scontrol execution failed

Caused by:
    Exit code: 1
    Stderr: slurm_load_jobs error: Invalid job id specified
    Stdout: ), "1739704": Err(scontrol execution failed

Caused by:
    Exit code: 1
    Stderr: slurm_load_jobs error: Invalid job id specified
    Stdout: ), "1739718": Err(scontrol execution failed
Caused by:
    Exit code: 1
    Stderr: slurm_load_jobs error: Invalid job id specified
    Stdout: ), "1739702": Err(scontrol execution failed

Caused by:
    Exit code: 1
    Stderr: slurm_load_jobs error: Invalid job id specified
    Stdout: ), "1739712": Err(scontrol execution failed

Caused by:
    Exit code: 1
    Stderr: slurm_load_jobs error: Invalid job id specified
    Stdout: ), "1739716": Err(scontrol execution failed

Caused by:
    Exit code: 1
    Stderr: slurm_load_jobs error: Invalid job id specified
    Stdout: )}))
[2024-10-01T06:07:58.812Z WARN  hyperqueue::server::autoalloc::process] Could not get status of allocation 1739716: scontrol execution failed

Caused by:
    Exit code: 1
    Stderr: slurm_load_jobs error: Invalid job id specified
    Stdout: 
[2024-10-01T06:07:58.812Z WARN  hyperqueue::server::autoalloc::process] Could not get status of allocation 1739715: scontrol execution failed

Caused by:
    Exit code: 1
    Stderr: slurm_load_jobs error: Invalid job id specified
    Stdout: 
[2024-10-01T06:07:58.812Z WARN  hyperqueue::server::autoalloc::process] Could not get status of allocation 1739718: scontrol execution failed

Caused by:
    Exit code: 1
    Stderr: slurm_load_jobs error: Invalid job id specified
    Stdout: 
[2024-10-01T06:07:58.812Z WARN  hyperqueue::server::autoalloc::process] Could not get status of allocation 1739717: scontrol execution failed

Caused by:
    Exit code: 1
    Stderr: slurm_load_jobs error: Invalid job id specified
    Stdout: 
[2024-10-01T06:07:58.812Z WARN  hyperqueue::server::autoalloc::process] Could not get status of allocation 1739711: scontrol execution failed

Caused by:
    Exit code: 1
    Stderr: slurm_load_jobs error: Invalid job id specified
    Stdout: 
[2024-10-01T06:07:58.812Z WARN  hyperqueue::server::autoalloc::process] Could not get status of allocation 1739702: scontrol execution failed

Caused by:
    Exit code: 1
    Stderr: slurm_load_jobs error: Invalid job id specified
    Stdout: 
[2024-10-01T06:07:58.812Z WARN  hyperqueue::server::autoalloc::process] Could not get status of allocation 1739704: scontrol execution failed

Caused by:
    Exit code: 1
    Stderr: slurm_load_jobs error: Invalid job id specified
    Stdout: 
[2024-10-01T06:07:58.812Z WARN  hyperqueue::server::autoalloc::process] Could not get status of allocation 1739710: scontrol execution failed

Caused by:
    Exit code: 1
    Stderr: slurm_load_jobs error: Invalid job id specified
    Stdout: 
[2024-10-01T06:07:58.812Z WARN  hyperqueue::server::autoalloc::process] Could not get status of allocation 1739700: scontrol execution failed

Caused by:
    Exit code: 1
    Stderr: slurm_load_jobs error: Invalid job id specified
    Stdout: 
[2024-10-01T06:07:58.812Z WARN  hyperqueue::server::autoalloc::process] Could not get status of allocation 1739712: scontrol execution failed

Caused by:
    Exit code: 1
    Stderr: slurm_load_jobs error: Invalid job id specified
    Stdout: 
[2024-10-01T06:07:58.812Z DEBUG hyperqueue::server::autoalloc::process] Task state: ServerTaskState { jobs: Map({JobId(4663): 1, JobId(4596): 1, JobId(4657): 1, JobId(4651): 1, JobId(4584): 1, JobId(4645): 1, JobId(4706): 1, JobId(4639): 1, JobId(4700): 1, JobId(4694): 1, JobId(4688): 1, JobId(4682): 1, JobId(4609): 1, JobId(4670): 1, JobId(4664): 1, JobId(4597): 1, JobId(4652): 1, JobId(4707): 1, JobId(4640): 1, JobId(4573): 1, JobId(4634): 1, JobId(4567): 1, JobId(4701): 1, JobId(4695): 1, JobId(4689): 1, JobId(4683): 1, JobId(4677): 1, JobId(4659): 1, JobId(4653): 1, JobId(4708): 1, JobId(4641): 1, JobId(4702): 1, JobId(4635): 1, JobId(4696): 1, JobId(4690): 1, JobId(4684): 1, JobId(4666): 1, JobId(4660): 1, JobId(4654): 1, JobId(4648): 1, JobId(4709): 1, JobId(4642): 1, JobId(4703): 1, JobId(4636): 1, JobId(4697): 1, JobId(4691): 1, JobId(4685): 1, JobId(4618): 1, JobId(4679): 1, JobId(4600): 1, JobId(4655): 1, JobId(4710): 1, JobId(4704): 1, JobId(4637): 1, JobId(4698): 1, JobId(4692): 1, JobId(4680): 1, JobId(4674): 1, JobId(4668): 1, JobId(4656): 1, JobId(4644): 1, JobId(4705): 1, JobId(4638): 1, JobId(4699): 1, JobId(4565): 1, JobId(4626): 1, JobId(4693): 1}) }, max. workers to spawn: 18446744073709551615

checking one job details (1739716), I see it quickly failed:

[svatosm@login3.karolina ~]$ sacct -j 1739716 --format=User,JobID,Jobname,partition,state,time,start,end,elapsed,MaxRss,MaxVMSize,Reqmem,nnodes,ncpus,nodelist
 User JobID           JobName  Partition      State  Timelimit               Start                 End    Elapsed     MaxRSS  MaxVMSize     ReqMem   NNodes      NCPUS        NodeList 
--------- ------------ ---------- ---------- ---------- ---------- ------------------- ------------------- ---------- ---------- ---------- ---------- -------- ---------- --------------- 
svatosm 1739716       hq-1-2221 qcpu_pree+     FAILED   12:00:00 2024-09-30T07:10:19 2024-09-30T07:10:20   00:00:01                             230G        1        128           cn123 
      1739716.bat+      batch                FAILED            2024-09-30T07:10:19 2024-09-30T07:10:20   00:00:01          0      2136K                   1        128           cn123 
      1739716.ext+     extern             COMPLETED            2024-09-30T07:10:19 2024-09-30T07:10:20   00:00:01          0        52K                   1        128           cn123

The reason why the job failed so fast (according to the stderr), were Permission Denied errors on /tmp (i.e. something wrong with the node). But this should not stop/block the submission forever. Creating a new allocation queue makes the submission start instantly.

It4innovations / hyperqueue

HyperQueue submission gets stuck if workers fail #764