Hackathon open issues

[dpryan79]

Below are open issues for our hackathon. Please feel encouraged to add to them:

Major

• [x] Conda directive in rules
  • [x] DNA: Vivek
  • [x] ChIP-seq: Devon
  • [x] ATAC-seq: Devon
  • [x] WGBS: Katarzyna
  • [x] HiC: Leily
  • [x] RNAseq: Steffen
  • [x] scRNAseq: Steffen
  • [x] Shared rules: Devon
• [x] Code cleanup, move common cmd args to parsercommon (like deeptools)
• [x] Install scripts for snakepipes basic (only snakemake, python, yaml etc) #110
• [x] Documentation #142
• [x] Small test datasets
• [x] more generic cluster config (aka hard coded SlurmEasy etc) see https://github.com/maxplanck-ie/snakepipes/pull/114

Other

• [x] STAR --limitBAMsortRAM option
  • [x] Alternatively, --outBAMsortingBinsN and version 2.6.0b. This is the suggested tweak from Alex
• [x] produce rule graph at each run
• [x] nicer logging #122
• [x] HiCExplorer hicSumMatrices based on samples description file, and subsequent hicMergeBins, hicCorrectMatrix, hicCallTADs, etc.
• [x] Multiple sample sheets for RNA-seq differential analysis
  • [x] update downsample fastq script to allow for /1 /2 differences in read name (see https://github.com/maxplanck-ie/snakepipes/issues/106)
  • [x] don't overwrite log files (https://github.com/maxplanck-ie/snakepipes/issues/58)
  • [x] Switch to Snakemake 5.0 or newer, which allows grouping rules and piping between them.
  • [x] 10X scRNA-seq workflow with cellranger. This isn't available in conda, so we should discuss whether it really gets merged in or not.
  • [x] Remove instances of writing to $TMPDIR and then moving the resulting files. This is pointless and prevents snakemake from cleaning up upon error.
  • [x] Re-evaluate target files. Things like trimmed fastq files should be temp()
  • [x] Update tool versions as appropriate.

Final items (i.e., those that need to be done once the branches are using conda):

• [x] Remove instances where CONDA_SHARED_ENV is set at the top of files, since this and similar global variables will now be set in the top-level snakefiles.

[gtrichard]

• [x] HiCExplorer hicSumMatrices based on samples description file, and subsequent hicMergeBins, hicCorrectMatrix, hicCallTADs, etc.

[mirax87]

• [x] Multiple sample sheets for RNA-seq differential analysis

[dpryan79]

@gtrichard @mirax87 I've updated my original post with your points, thanks!

[vivekbhr]

I added the Major issues to fix

[dpryan79]

Just a note that the conda environments for the workflows should be versioned. The environment version doesn't need to track the snakepipes version, since updating packages and updating the workflow are different tasks.

[steffenheyne]

Afaics, the pointless TMPDIR usage is only in TrimGalore.yaml where trimming first writes to TMPDIR and then we move the file back to the snakemake folder...the other places where we use TMPDIR, we use it more as "real" temp path like in samtools -T ...and I think this usage is good and useful!

[dpryan79]

The samtools $TMPDIR usage is totally legit :)

[vivekbhr]

@gtrichard we decided not to implement sample-sheets in the HiCEx pipeline, since the thing you mentioned can simply be done by splitting the samples into sets and running the pipeline for each set.