Batch mode execution

[apdavison]

In scientific computing it is common to run a batch of computations with the same code, but different parameters on each run. It would be useful for Sumatra to support this, perhaps by introducing a new command, smt batch.

It would be helpful here to collect different ways in which such batch computations are currently specified.

• a shell script is used to loop over different parameter combinations
• the parameters package provides a "parameter space" concept, which enables specifying parameter combinations easily in a JSON-like format.

• command-line tools can take multiple values for each parameter, where each combination of parameters represents a new execution (see #208).

  Imported from Bitbucket issue 209, created by apdavison on 05-12-2014 at 08:07, last modified: 05-13-2014 at 11:42

[apdavison]

Two random brainstorming comments:

• Does Lancet [1] or its underlying Param package [2] have anything to offer in this regard (either for inspiration, or to outsource functionality)? I haven't actually worked with Lancet yet, but it appears to have relatively sophisticated support for exploring parameter spaces. Unfortunately, the documentation seems a bit sparse.
• It would be great to have some support in Sumatra for analyzing the outcomes of batch computations. This ties in with filtering records (e.g. based on parameters values) and better pandas integration. I'm currently exploring some workflows for myself, but it might be good to discuss this on the mailing list at some point to get an idea of other people's workflows and what's needed in Sumatra vs. what can be outsourced to other packages.

[1] http://ioam.github.io/lancet/

[2] https://github.com/ioam/param

Trac comment by cilix on 05-13-2014 at 11:42

[sjdenny]

I have used Lancet successfully for performing parameter sweeps on a compute cluster. For example, a grid search can be specified as a product of individual parameter ranges, then each simulation is submitted individually to qsub. When the simulations finish, Lancet has a nice way of specifying the output file range and running a user-defined function to amalgamate the results. The paper [1] was useful for getting a working example together, although it seems that some of the examples there have now made it through to the documentation.

I'd be extremely interested to see if sumatra and Lancet (or something Lancet-like) could be combined, since in our work, one 'experiment' is usually hundreds of related cluster jobs which are then combined and analysed together.

[1] http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3874632/