Chunking and inner parallelization

[rrichmond]

Is there a simple way to have a slave run the jobs submitted to it in a chunk in parallel? Right now I'm passing the algorithm parameter as a list and then iterating over that in parallel. I thought this might be possible given the discussion in submitJobs, but I can't seem to find a way to get the job parameters to make this work.

[mllg]

Yes, there is. You can use parallelMap or foreach to parallelize the execution of the chunks on a node. See https://mllg.github.io/batchtools/reference/submitJobs.html#inner-parallelization. Basically, you pass down some resources to submitJobs(). batchtools will then setup the respective backend for parallelMap or foreach(). Also note the last few lines of the example of submitJobs().

[mllg]

You still would have to map over the passed list of parameters to get the parallelization though.

There was a version of batchtools which parallelized the execution of chunks, but this was quite error-prone w.r.t. capturing the output.

[rrichmond]

Great, thanks!