(DSL2) Methyldactyl extract does not produce expected bedGraph when using --methylkit param

TheodoreMarkulin commented 2 years ago

Description of the bug

In the DSL2 branch, when using the --methylkit parameter in conjunction with methyldactyl the pipeline fails. This appears to be due to an 'either or' kind of behavior wherein Methyldactyl extract will produce *_CpG.methylKit with --methylkit used or `*.bedGraph without it used but both are never produced from this step.

The step currently fails because it expects a *.bedGraph file. MultiQC also expects the *.bedGraph file to be produced.

Steps to reproduce

Run any version of the pipeline within DSL2 with methldactyl with the --methyl_kit parameter set

Expected behaviour

Production of the *.methyKit files,
Completion of the pipeline

Command Used

nextflow run $PIPELINE -with-tower '<IP ADDRESS>'  -bucket-dir $BUCKET_DIR  --input $DATA --outdir $OUTPATH_PREFIX/methyldactyl --fasta $REFERENCE --aligner bwameth --clip_r1 20 --clip_r2 15 -c "$CONFIG" --methyl_kit

System

Hardware: Amazon
Executor: AWS Batch
OS: Amazon Linux

Nextflow Installation

Version: 21.10.06
Pipeline Version: dsl2 branch, commit- ba50ea9ce3c4c5aea0504d6c4aa07b8516335421

Container engine
Engine: Docker
version:
Image tag: quay.io/biocontainers/methyldackel:0.6.0--h22771d5_0

ewels commented 2 years ago

Replicated using GitPod on the dsl2 branch:

gitpod@/workspace/methylseq/testing(dsl2):$ nextflow run ../ --aligner bwameth --methyl_kit -profile test,docker --outdir ./results/ 

Picked up JAVA_TOOL_OPTIONS:  -Xmx3489m
N E X T F L O W  ~  version 22.04.0
Launching `../main.nf` [distraught_lovelace] DSL2 - revision: 071d41655d

------------------------------------------------------
                                        ,--./,-.
        ___     __   __   __   ___     /,-._.--~'
  |\ | |__  __ /  ` /  \ |__) |__         }  {
  | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                        `._,._,'
  nf-core/methylseq v2.0dev
------------------------------------------------------
Core Nextflow options
  runName                   : distraught_lovelace
  containerEngine           : docker
  launchDir                 : /workspace/methylseq/testing
  workDir                   : /workspace/methylseq/testing/work
  projectDir                : /workspace/methylseq
  userName                  : gitpod
  profile                   : test,docker
  configFiles               : /workspace/methylseq/nextflow.config

Input/output options
  input                     : https://raw.githubusercontent.com/nf-core/test-datasets/methylseq/samplesheet/samplesheet_test.csv
  outdir                    : ./results/

Reference genome options
  fasta                     : https://github.com/nf-core/test-datasets/raw/methylseq/reference/genome.fa
  fasta_index               : https://github.com/nf-core/test-datasets/raw/methylseq/reference/genome.fa.fai

Alignment options
  aligner                   : bwameth

bwa-meth options
  methyl_kit                : true

Max job request options
  max_cpus                  : 2
  max_memory                : 6 GB
  max_time                  : 6h

Institutional config options
  config_profile_name       : Test profile
  config_profile_description: Minimal test dataset to check pipeline function

!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use nf-core/methylseq for your analysis please cite:

* The pipeline
  https://doi.org/10.5281/zenodo.1343417

* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x

* Software dependencies
  https://github.com/nf-core/methylseq/blob/master/CITATIONS.md
------------------------------------------------------
executor >  local (41)
executor >  local (41)
[9e/f95620] process > NFCORE_METHYLSEQ:METHYLSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (samplesheet_test.csv)    [100%] 1 of 1 ✔
[ef/e4b84b] process > NFCORE_METHYLSEQ:METHYLSEQ:FASTQC (SRR389222_sub2_T1)                              [100%] 3 of 3 ✔
[df/568fdd] process > NFCORE_METHYLSEQ:METHYLSEQ:TRIMGALORE (SRR389222_sub2_T1)                          [100%] 3 of 3 ✔
[f4/cba01c] process > NFCORE_METHYLSEQ:METHYLSEQ:ALIGNER:BWAMETH_INDEX (bwameth/genome.fa)               [100%] 1 of 1 ✔
[0a/e98d4a] process > NFCORE_METHYLSEQ:METHYLSEQ:ALIGNER:BWAMETH_ALIGN (SRR389222_sub2_T1)               [100%] 3 of 3 ✔
[b9/a4c670] process > NFCORE_METHYLSEQ:METHYLSEQ:ALIGNER:SAMTOOLS_SORT (SRR389222_sub2_T1)               [100%] 3 of 3 ✔
[19/ccb06d] process > NFCORE_METHYLSEQ:METHYLSEQ:ALIGNER:SAMTOOLS_INDEX_ALIGNMENTS (SRR389222_sub2_T1)   [100%] 3 of 3 ✔
[19/3b273c] process > NFCORE_METHYLSEQ:METHYLSEQ:ALIGNER:SAMTOOLS_FLAGSTAT (SRR389222_sub2_T1)           [100%] 3 of 3 ✔
[01/ba278d] process > NFCORE_METHYLSEQ:METHYLSEQ:ALIGNER:SAMTOOLS_STATS (SRR389222_sub2_T1)              [100%] 3 of 3 ✔
[37/6aea49] process > NFCORE_METHYLSEQ:METHYLSEQ:ALIGNER:PICARD_MARKDUPLICATES (SRR389222_sub2_T1)       [100%] 3 of 3 ✔
[6d/41715c] process > NFCORE_METHYLSEQ:METHYLSEQ:ALIGNER:SAMTOOLS_INDEX_DEDUPLICATED (SRR389222_sub2_T1) [100%] 3 of 3 ✔
[49/09821b] process > NFCORE_METHYLSEQ:METHYLSEQ:ALIGNER:METHYLDACKEL_EXTRACT (SRR389222_sub3_T1)        [100%] 3 of 3, failed: 3 ✘
[3b/484584] process > NFCORE_METHYLSEQ:METHYLSEQ:ALIGNER:METHYLDACKEL_MBIAS (SRR389222_sub1_T1)          [100%] 3 of 3 ✔
[63/02a495] process > NFCORE_METHYLSEQ:METHYLSEQ:QUALIMAP_BAMQC (SRR389222_sub1_T1)                      [100%] 3 of 3 ✔
[65/255a68] process > NFCORE_METHYLSEQ:METHYLSEQ:PRESEQ_LCEXTRAP (SRR389222_sub2_T1)                     [100%] 3 of 3, failed: 3 ✔
[-        ] process > NFCORE_METHYLSEQ:METHYLSEQ:CUSTOM_DUMPSOFTWAREVERSIONS                             -
[-        ] process > NFCORE_METHYLSEQ:METHYLSEQ:MULTIQC                                                 -
Execution cancelled -- Finishing pending tasks before exit
-[nf-core/methylseq] Pipeline completed with errors-
Error executing process > 'NFCORE_METHYLSEQ:METHYLSEQ:ALIGNER:METHYLDACKEL_EXTRACT (SRR389222_sub2_T1)'

Caused by:
  Missing output file(s) `*.bedGraph` expected by process `NFCORE_METHYLSEQ:METHYLSEQ:ALIGNER:METHYLDACKEL_EXTRACT (SRR389222_sub2_T1)`

Command executed:

  MethylDackel extract \
      --methylKit \
      genome.fa \
      SRR389222_sub2_T1.markdup.sorted.bam

  cat <<-END_VERSIONS > versions.yml
  "NFCORE_METHYLSEQ:METHYLSEQ:ALIGNER:METHYLDACKEL_EXTRACT":
      methyldackel: $(MethylDackel --version 2>&1 | cut -f1 -d" ")
  END_VERSIONS

Command exit status:
  0

Command output:
  (empty)

Command error:
  Unable to find image 'quay.io/biocontainers/methyldackel:0.6.0--h22771d5_0' locally
  0.6.0--h22771d5_0: Pulling from biocontainers/methyldackel
  c1a16a04cedd: Already exists
  4ca545ee6d5d: Already exists
  d0ae09c59d4e: Pulling fs layer
  d0ae09c59d4e: Verifying Checksum
  d0ae09c59d4e: Download complete
  d0ae09c59d4e: Pull complete
  Digest: sha256:adfbe21e9ac53d92627bde7b5ebe33180ae8ed90f479c63940742731666a76f4
  Status: Downloaded newer image for quay.io/biocontainers/methyldackel:0.6.0--h22771d5_0
  writing to prefix:'SRR389222_sub2_T1.markdup.sorted'

Work dir:
  /workspace/methylseq/testing/work/dd/738596c39359a76e54d75d73f3b902

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

ewels commented 2 years ago

Fixed 👍🏻

nf-core / methylseq