I've been troubleshooting stability of batchtools when used on Slurm with the default makeClusterFunctionsSlurm (PR #276 & #277 ).
The last (rare) error I can reproduce is:
Expected Behaviour
If a submitted job is requeued by Slurm:
batchtools should not report an expired status -> #277.
If the job should have run without error at first submission, the requeued job should also run successfully (assuming no fatal hardware errors)
Problem
Slurm jobs which are requeued because of a previous hardware failure fail within 30 seconds of starting the second run.
Reprex
Awkward, because it relies on an available (non-mission-critical) Slurm cluster, but manually deleting the worker node (via GCP) of a running & error-free job results in a requeue, a delay, then a reliable error about 20 seconds after the job begins its second run (file path removed for posting):
Error in gzfile(file, "rb") : cannot open the connection
Calls: <Anonymous> -> doJobCollection.character -> readRDS -> gzfile
In addition: Warning message:
In gzfile(file, "rb") :
cannot open compressed file '.../jobs/job929872958e6074e5662a4c9hd3f312f4.rds', probable reason 'No such file or directory'
Cause
batchtools:::doJobCollection.character deletes the jobCollection file.rds on the first run, so when the failed job gets requeued the file is no longer there, causing the error.
Workaround
Passing chunks.as.arrayjobs = TRUE in the resources request prevents this error (even if jobs are submitted singly) as it prevents the first run of the job deleting the jobCollection .RDS.
Questions
Apart from needing to clean up the files afterwards, can you see any downsides of using chunks.as.arrayjobs = TRUE for single jobs too? If not, this could be a useful default setting for @HenrikBengtsson when submitting jobs from future.batchtools, simply to avoid triggering an unhandled error, and to allow jobs to requeue as expected (assuming backend configuration allows).
Perhaps a more explicit option would be better - allow.requeue or prevent.requeue?
stuvet
commented
3 years ago
I've been troubleshooting stability of
batchtoolswhen used on Slurm with the defaultmakeClusterFunctionsSlurm(PR #276 & #277 ).The last (rare) error I can reproduce is:
Expected Behaviour
batchtoolsshould not report an expired status -> #277.Problem
Reprex
Cause
batchtools:::doJobCollection.characterdeletes the jobCollection file.rds on the first run, so when the failed job gets requeued the file is no longer there, causing the error.Workaround
chunks.as.arrayjobs = TRUEin the resources request prevents this error (even if jobs are submitted singly) as it prevents the first run of the job deleting the jobCollection .RDS.Questions
chunks.as.arrayjobs = TRUEfor single jobs too? If not, this could be a useful default setting for @HenrikBengtsson when submitting jobs fromfuture.batchtools, simply to avoid triggering an unhandled error, and to allow jobs to requeue as expected (assuming backend configuration allows).allow.requeueorprevent.requeue?