Closed wlandau closed 6 years ago
This looks like a batchtools
problem, but it could also have something to do with the future
ecosystem. I am not sure. cc @HenrikBengtsson.
Is this reproducible? Could be a problem with file system latency.
NB: I've improved the error message in 804dcc3b3dc870b9bc0ef5ed42d6cf90c89fc3b5
c97672eb12ad9743988a55637007f503a03e6be0 eventually solves this if this is an issue with the file system latency.
I do not know if the same thing happens on other machines. I only have access to my company's SGE cluster, and the problem persists even after updating to https://github.com/mllg/batchtools/commit/c97672eb12ad9743988a55637007f503a03e6be0.
With the latest commit you should get a more informative error message. Can you check if the reported log file exists on the file system?
Yes, I do see a more informative error message after updating batchtools
to the latest commit.
library(future.apply)
library(future.batchtools)
plan(batchtools_sge, template = "sge-simple.tmpl")
future_lapply(1:2, sqrt)
Error: Log file '/home/<CENSORED>/scratch/.future/20180607_111654-OPL6Ge/batchtools_128392604/logs/jobafa83af0c7534ee2be6cb550d768bbad.log' for job with id 1 not available
And sure enough, the folder .future/20180607_111654-OPL6Ge/batchtools_128392604/logs
exists, but it is empty.
SGE did write log files (job4b1879cb08606227955b9beb24628cdb.job.o29021485
and jobafa83af0c7534ee2be6cb550d768bbad.job.o29021482
) to the working directory of the R session, and those look fine.
### [bt]: This is batchtools v0.9.10.9000
### [bt]: Starting calculation of 1 jobs
### [bt]: Setting working directory to '/home/<CENSORED>/scratch'
### [bt]: Memory measurement disabled
### [bt]: Starting job [batchtools job.id=1]
### [bt]: Setting seed to 1 ...
### [bt]: Job terminated successfully [batchtools job.id=1]
### [bt]: Calculation finished!
The traceback in the R session is basically the same:
> traceback()
16: stop(simpleError(sprintf(...), call = NULL))
15: stopf("Log file '%s' for job with id %i not available", log.file,
id$job.id)
14: readLog(id, reg = reg)
13: extractLog(readLog(id, reg = reg), id)
12: getLog(id = jobid, reg = reg)
11: await.BatchtoolsFuture(future, cleanup = FALSE)
10: await(future, cleanup = FALSE)
9: value.BatchtoolsFuture(tmp, ...)
8: value(tmp, ...)
7: values.list(fs)
6: values(fs)
5: future_lapply(1:2, sqrt) at reprex.R#5
4: eval(ei, envir)
3: eval(ei, envir)
2: withVisible(eval(ei, envir))
1: source("reprex.R")
Ok, we are getting there...
SGE did write log files (job4b1879cb08606227955b9beb24628cdb.job.o29021485 and jobafa83af0c7534ee2be6cb550d768bbad.job.o29021482) to the working directory of the R session, and those look fine.
This is the wrong directory. You need the following line in your template to redirect the output (c.f. https://github.com/mllg/batchtools/blob/master/inst/templates/sge-simple.tmpl):
#$ -j y
#$ -o <%= log.file %>
It worked, thanks!
Does the following look right to you for SLURM? I do not have access to SLURM, so I cannot test it.
I recently started encountering errors submitting to an SGE cluster via
future.batchtools
,future.apply
, and this template file.