pipelines implemented with a pipeline framework

[pdp10]

In the future it might be worth investigating how to implement sbpipe pipelines using a pipeline framework:

• Ruffus <- elegant, cluster support seems a nightmare though..
• Luigi <- seems quite easy to use, the scheduling model might not be great for what sbpipe needs though!
• Toil <- powerful, supports SGE, Common Workflow Language (CWL) support. Provided examples don't even run.....their hello world fails..
• Snakemake <- make-like. Supports SGE. Needs sbpipe to be converted to python scripts.
• Pegasus
• Taverna
• Galaxy
• BPipe (bash - discard)
• etc.

If this transition will be implemented a few important points will have to be addressed first:

• current input parameters
• parallel execution using SGE or LSF (<= this is fundamental for sbpipe)
• 1:1, 1:n, n:1 transformations
• regular expressions to be implemented
• ...

Considering this review (http://oup.silverchair-cdn.com/oup/backfile/Content_public/Journal/bib/PAP/10.1093_bib_bbw020/2/bbw020.pdf), it seems to me that an explicit syntax, and class or configuration paradigm is the way to go..

NOTES:

• ruffus: one main developer. Not much work on it recently..
• toil: a few developers. Currently working on it. (BD2K Center for Translational Genomics)
• luigi: a few developers. Currently under development. No dev holes. Pretty good, but the scheduling model might not be great for what sbpipe needs (http://luigi.readthedocs.io/en/stable/execution_model.html).

[pdp10]

This could be the next version of sbpipe.

Snakemake works quite well for this. It supports SGE too. It requires the conversion of Python code to scripts. It is very flexible though. This can be an advantage in the long term..

[pdp10]

Pilot study

• Tested the ps2 pipeline using Snakemake (report generation is incomplete, but all the previous part works) (snakemake -s FILENAME).
• A way to pass sbpipe configurations to this new pipeline is still missing.
• Path files can be a bit tricky. It seems that Snakemake requires the full path to the r script.
• The location of the pipeline can works fine even if the test model is located somewhere else.

I need to test how parallel computation works with this.

[pdp10]

To test:

• yaml configuration files for snakemake

[pdp10]

To test:

• parallel computation.

[pdp10]

To test:

• log files.