An alternative to the current linear scheduler

[willu47]

Is your feature request related to a problem? Please describe. The current naive scheduler implementation is limited to linear jobs. It would be great to allow parallel paths to be exploited on a laptop, but allow scalability to cluster, or the cloud.

Describe the solution you'd like Snakemake provides all of these features. Smif could write a snakemake configuration file from the system of systems configuration, which could then be deployed on a laptop or cluster. Snakemake also allows each "rule" within a snakemake workflow to be run within a specific conda environment, or on a container, enabling good scalability.

Describe alternatives you've considered There are many alternative workflow management systems, but this seems a good candidate.

Additional context Snakemake has an experimental gui, and can render images of a workflow using dot. It's written in Pure python. It can be pip installed, or from the bioconda channel.

[tomalrussell]

I like the idea of getting smif clearly out of the scheduler business :slightly_smiling_face:

In the meantime, here's a practical way to run a stack of smif model runs in parallel, which should tackle one variant of the problem (but won't help with running jobs in parallel within a single model run):

cat batchfile | parallel -j 20 smif run {} -v -i local_binary

Where batchfile is a text file containing a model run name on each line. smif could run this sequentially using smif -b batchfile. Instead, piping it into parallel means each model run will run concurrently, up to the limit optionally passed in with -j (jobs).