carpentries-incubator / snakemake-novice-bioinformatics

Introduction to Snakemake for Bioinformatics
https://carpentries-incubator.github.io/snakemake-novice-bioinformatics
Other
18 stars 9 forks source link

Avoid uncontrolled temporary files, and explain why #29

Closed tbooth closed 2 years ago

tbooth commented 2 years ago

From Lana Talmane:

I am generally not a fan of using temporary files that are not defines within snakemake rules and inputs/outputs and just appear in the shell command. I just don’t trust myself to not mess it up at some point by overwriting some files. Like in the example Lesson 6 Option 4 solution would be overwriting the same .html and .zip files for the trimmed and untrimmed reads. Because the rule is clearly written to be generalizable for different {indir} (trimmed/reads), this is a bit of a dangerous example. But this makes for a good question for the learners, as to why this might be dangerous,

tbooth commented 2 years ago

Agreed, and I've now added a note about this issue, which is a common gotcha. However, I think the correct option here is to use shadow: "minimal" which makes the problem go away, at the expense of making debugging trickier. I've mentioned shadow rules in ep12 but I think I've undersold them, so I've changed the text in ep12 and referred back to ep06