Add ability to run a list of parameter sets in parallel

[roseDickinson]

We currently have a feature that will allow you to loop over a set of parameters and values in Workflows, that will work similar to a "while" or a "for" loop. It would almost certainly run each step of the loop in parallel (unless a step required the output of the previous step). Would that help you in this case? The issue associated with this work is CORE-279

[fmcclean]

Yes that sounds like what we need. I'd be interested to hear how you are planning to handle the output data from these multi-parameter runs i.e. will new datasets be created for each step or will they be combined into a single dataset and if so how will this be structured.

[tomalrussell]

Just to support this - this will be crucial for big scenario/ensemble/sensitivity analysis runs.

Related to the question of how output data is handled - it would typically be useful to have some kind of aggregation step run after the loop. At a high level this looks a bit like map-reduce, mapping simulation runs over a list of parameter sets, then doing some stats over the outputs. Is that something that CORE-279 would enable?

[Craig-Robson]

Joining the previous comments, this is essential for a lot of modelling scenarios, and I think a must and a great selling point when talking about DAFNI and the modelling service - I'm not sure that many other currently available services offer such support. Important to make sure this is enabled correctly in terms of how a user can set this up so not too complicated.

As also mentioned previously, not sure how outputs from such runs should be handled and stored, but can see maybe the easiest solution being individual sub-directories for each run with a contents file to link each result sub-directory to parameter values.