garlick
commented
2 weeks ago Slow iscsi root has been blamed for similar problems lately. Agreed, a more generous default timeout makes sense here.
Closed grondo closed 2 weeks ago
garlick
commented
2 weeks ago Slow iscsi root has been blamed for similar problems lately. Agreed, a more generous default timeout makes sense here.
In testing on fluke, I submitted a slew of jobs and then canceled them, to test cancellation of jobs while the prolog is running. A small number of nodes were drained with "prolog canceled then timed out". Here's relevant logs from an affected job and node:
The prolog is started at 15:52:53, then canceled at 15:53:27 (
flux job eventlogsadly doesn't have a way to present timestamps in the current timezone, will have to fix, so I'm converting to PST here)The prolog systemd unit shows as started at 15:52:57. A bit unexpected and concerning that it took 5s to just start the unit :thinking:
and the unit got
Main process exitedby 15:53:42 .. that's a delay of 15s between the cancel exception and the prolog actually being killed:By this time, the default 10s prolog cancel timeout had elapsed, a job exception was raised the node drained:
It then took another 11s before the unit failed and the prolog was stopped.
If delays like this are common with the systemd managed prolog, it might be wise to increase the SIGTERM timeout from 10s to something more like 30s or 1m. This can be done via configuration, but perhaps a new default would make sense.