stuvet
commented
3 years ago This seems to be caused by future.batchtools calling batchtools::getStatus, which defaults to 'expired' if is.na(status). It doesn't affect batchtools as it's waiting for the log to appear in waitForFile.
Here's an example output of batchtools::getStatus while the worker is provisioning (& batchtools is in the waitForFile loop):
## result of batcthools::getStatus
Status for 1 jobs at 2021-06-05 09:20:42:
Submitted : 1 (100.0%)
-- Queued : 0 ( 0.0%)
-- Started : 0 ( 0.0%)
---- Running : 0 ( 0.0%)
---- Done : 0 ( 0.0%)
---- Error : 0 ( 0.0%)
---- Expired : 1 (100.0%)
## The reg entry during provisioning
batch.id job.id def.id submitted started done error mem.used resource.id
1: 5955 1 1 1622885612 NA NA <NA> NA 1
log.file job.hash job.name status
1: <NA> jobc59eb7d6a4f24ec81611dd0a592cb9de todo_chains_2e8f2ae4 <NA>Perhaps an explicit call to batchtools::waitForFile in future.batchtools (linked to an option to set fs.latency from #73?) could be a workaround for this, though I'll take a closer look at batchtools to see if I can adapt the getStatus result for this case - it may be possible to differentiate 'provisioning' from 'expired' if the log.file is populated in the reg entry, though this strategy may cause jobs to run indefinitely if they expire before the log file is produced?
coatless
I have submitted a simple pull request for a bugfix in
batchtoolswhich fuelled some of its behaviour mentioned in #73.I'm writing here because this proposed bugfix reveals an error in future.batchtools, though based on the future.debug output I don't believe the two are related - just that the batchtools bug previously threw the error first.
Describe the bug When a slurm worker is availble for the job, (& when the batchtools::waitForFile is not required by batchtools::getLog -> batchtools::readLog), everything functions correctly (note the status):
But when a slurm worker needs to be provisioned to run the job (& so the batchtools::waitForFile will also be called), the initial result of the call to
future.batchtools::status(future)infuture.batchtools::awaitis incorrect:At this point, logging inserted into
batchtools:::waitForFilebegins to appear. No more future.debug messages appear until the log file has been detected (now, after the proposed bugfix) &batchtools::waitForFileexits.To be clear, the logged output does exist, and continues to be written by the running job after future.batchtools::await flags the job as expired & exits.
Please let me know if there's anything else I can do to help resolve this issue.
Expected behavior future.batchtools::await waits for the running job to exit, even when workers need to be provisioned or batchtools::waitForFile is triggered.
Session information