Closed smilesun closed 5 years ago
The column status
is part of the data.table batch.ids
which is returned by getBatchIds()
. getBatchIds()
internally queries the scheduler for running jobs which in your case seems to be failing. You could try debugging
reg$cluster.functions$listJobsRunning(reg)
which should return process IDs in multicore mode.
The jobs in the last table are detected as expired because:
for me the expired jobs seem to happen when there is not enough RAM allocated per job -- increasing the memory resources seems to fix the issue.
for me the expired jobs seem to happen when there is not enough RAM allocated per job -- increasing the memory resources seems to fix the issue.
The reported status then seems correct to me. The jobs started, but did not communicate back any results because they were killed by the scheduler as they consumed too much memory.
A job counts as expired, iff it has been submitted, but it is not terminated (results/errors are written to the file system) AND is not on the system any more.
Also note that jobs may seem expired temporarily. On some systems, under heavy load, it takes more than 60s after a job is terminated before the files written by a job can be found on the master. In these cases, you just need to be patient.
Hope this helps.
maybe it is a documentation issue. I did not understand what expired means, but your description helps a lot. I would suggest adding some definition of "expired" on https://mllg.github.io/batchtools/articles/batchtools.html which currently does not explain it at all.
Hi, I do not quite understand the following code, each time I submit jobs either using slurm or makeClusterFunctionsMulticore() , running jobs are always shown as expired.
if I debug this function, and run
reg$status
, it showedIn the following code, I do not understand where is the status in
is.na(status)
For the merged table
res = merge(filter(reg$status, ids), batch.ids, by = "batch.id", all.x = TRUE, all.y = FALSE, sort = FALSE)
It showed