Closed t-reents closed 1 month ago
Hi, when you cancel a HQ job, the following happens:
That's about all that HQ can do from user-space. If the srun
job step does not react to these two signals (and indeed from my experience, Slurm job steps can be very.. finicky around signals), then it is possible that the process will continue running, even though HQ no longer treats it as running.
In general, I would suggest to avoid running srun
inside the submitted HQ tasks, unless you specifically need it for multi-node MPI execution.
Hi @Kobzol! Thanks a lot for your quick reply!
I was already expecting something like that. Unfortunately, we need this multi-node use-case. Nonetheless, thank you for the clarification.
I made the observation that after calling
hq job cancel <id>
the job doesn't seem to be properly canceled. For this example, I manually start a worker by submitting a slurm submission script which specifies the allocation:The approach works per se and HQ runs perfectly fine (also my submissions within HQ). The actual jobs are submitted via the following script using
hq job submit
:(I need to use srun, similar to #443, and the problem occurs independent of the chosen srun options). In general, running the jobs like this works well. However, when cancelling the job via
hq job cancel
, the slurm job steps are still listed as running:Moreover, when resubmitting the HQ job, I observe a worse performance, as the allocation still seems to be busy. The following output is generated after resubmission:
If I don't use the
--overlap
option in the srun command, I'll also receive thesrun: Job 7693701 step creation temporarily disabled, retrying (Requested nodes are busy)
warning/error (similar to #443) which kind of confirms that the previous steps are not fully cancelled.Any help would be much appreciated!