Make jobname match number in batchtools_XXXXXXXXXX by default when using SLURM?

[kendonB]

When I want to peek at what's going on on a future-powered slurm job I look at, for example:

.future/20171118_083108-ebBlfz/batchtools_492448437/logs/job9622adcaff15f092fcb3ed1f0d6221bd.log

I have to get in to the final filename to match up the job (here job9622ad) to a log. It would be a little easier to get to the job I want if the jobname was also after the batchtools_ bit. Is there a reason this doesn't make sense?

[HenrikBengtsson]

The format of that path is: .future/<timestamp>_{{R-SESSION-ID}}/batchtools_{{FUTURE_ID}}/logs/job{{BATCHTOOLS-JOB-ID}}.log

I agree that having ../batchtools_{{FUTURE_ID}}_{{BATCHTOOLS-JOB-ID}}/... instead of ../batchtools_{{FUTURE_ID}}/... could be handy. Unfortunately, I'm not sure this can be done, because that batchtools_{{FUTURE_ID}} part must be set up before submitting a batchtools job (because that is actually the batchtools registry name). Only later is the {{BATCHTOOLS-JOB-ID}} created when the job is created/submitted.

When you say you "want to peek", do you mean from outside of R while the job is running? That is, it would not be helpful to provide accessor functions to the BatchtoolsFuture class?

[kendonB]

I have been looking from outside of R while the job is running, yes. However, if the best way to achieve the peeking would be from within a second R session, then, of course, that would also work great.

[HenrikBengtsson]

I do that at times as well. Also, when there's an error, I find myself navigating these temporary future+batchtools repositories by hand from within R (the default behavior of future.batchtools is to not clean them up if there's an error).

Having an R API to peek into/browse around in an existing (and possibly even a live/hot) batchtools job registry would be neat. If possible, it's not hard to imagine of pulling out an overview of current jobs, e.g. classical stdout, fancy color-annotated stdout, or even a GUI/Shiny dashboard. I think that such a feature request for a basic querying API is better address to the batchtools project itself.

[kendonB]

Ref: https://github.com/mllg/batchtools/issues/157

[HenrikBengtsson]

Note to self: good news from https://github.com/mllg/batchtools/issues/157#issuecomment-345954380:

To clarify this feature request: Is it possible (and safe) to set up a pointer to a batchtools registry from a separate R session (by only knowing its path on the file system) and use the find*() functions to query it's status?

Yes, it should be possible if you use loadRegistry("<path>", writeable = FALSE). If you encounter problems while only monitoring, it is a bug in batchtools.