BatchJobs sometimes returns Error syncing registry... database is locked

[alexvorobiev]

There is a corresponding issue https://github.com/tudo-r/BatchJobs/issues/58 and one solution seems to be to add job.delay parameter to submitJobs because very short jobs can outperform the NFS (un)locking the database file. Where does it fit in future.BatchJobs API?

[HenrikBengtsson]

I'll see how I can add this feature. Maybe something like plan(batchjobs_slurm, resources=..., job.delay=...).

[HenrikBengtsson]

I'm going to add some kind of support in the future package for users to control BatchJobs::submitJobs() arguments job.delay and max.retries. The cleanest would to have these set once by the user, i.e. not per call. For instance, they could be properties of the registry or even global options. But that's not how BatchJobs was implemented. So, I'm leaning to either:

1. specify them via future::plan(batchjobs_local, job.delay=...), or
2. set them up as options(future.BatchJobs.submitJobs.job.delay=...)

The first approach is how resources are already specified, so I'm leaning towards that.

@mllg, I was peeking at batchtools to see what you're using going forward. However, batchtools::submitJobs() doesn't take those arguments;

> args(batchtools::submitJobs)
function (ids = NULL, resources = list(), reg = getDefaultRegistry()) 

Are settings such as job.delay somewhere else in batchtools, no longer used, or are you planning to add them later? I'm asking because I'd like to make future.BatchJobs to be as similar as possible to future.batchtools (to be implemented) already now.

[mllg]

@mllg, I was peeking at batchtools to see what you're using going forward. However, batchtools::submitJobs() doesn't take those arguments;

Yes you're right.

• job.delay: No database which can be locked, therefore not needed anymore
• max.retries: Never touched this myself, don't know if someone ever did. If this will get reimplemented, it will be a resource and you don't have to worry about it (?).

[HenrikBengtsson]

Thanks for this; it helps my planning. I'll probably go forward by adding support for future::plan(batchjobs_local, job.delay=...).

[HenrikBengtsson]

@alexvorobiev, please try the develop version;

Version: 0.13.0-9000 [2016-09-01]
o Added argument 'job.delay' to batchjobs_*() functions for passing
  it as is to BatchJobs::submitJobs() used when launching futures.

You can install it as:

source('http://callr.org/install#HenrikBengtsson/future.BatchJobs@develop')

[alexvorobiev]

I tried it and didn't notice any issues.

[HenrikBengtsson]

@alexvorobiev, just so I don't misunderstand; you're saying this update works for you? Can we close this issue?

[alexvorobiev]

Yes, the issue can be closed.