Discuss limitations & alternatives to Snakemake

[tkphd]

Snakemake may not be the best workflow manager for HPC; we're teaching it because it is broadly accessible, and the lessons can be broadly transferrable.

Include an episode at the end to discuss the limitations of Snakemake, and introduce alternative tools (Parsl, Fireworks) for HPC.

[ocaisa]

There's a useful flow chart from CSC to guide you through workflow tools recommendations based on your use case.

[bkmgit]

Can look at this.

[bkmgit]

One alternative is the Concise Common Workflow Language which seems to also be used in bioinformatics , an overview is available in this Guix HPC blog post. The CSC guide above seems to indicate that Conda is not great for HPC, and should be packaged. It does recommend GNU parallel for some use cases. @mikerenfro @wirawan0 comments welcome

[bkmgit]

Related work:

• https://github.com/biosustain/snakemake_UmetaFlow
• https://www.kitware.com/hpccloud-hpc-simulation-workflow-in-the-cloud/
• https://pdc-support.github.io/hpc-intro/12-workflows/