issue with multiqc

[sunta3iouxos]

this is this is the error code when reaching the multifastq, this is the same pipeline as #1021
I removed some entries to make it readable. take note the version of my snakePipes

$snakePipes version
version 2.8.0

TMPDIR=/scratch/tgeorgom/temp/ UTEMP=$(mktemp -d ${TMPDIR:-/tmp}/snakepipes.XXXXXXXXXX); XDG_CACHE_HOME=$UTEMP TMPDIR=/scratch/tgeorgom/temp/ PYTHONNOUSERSITE=True snakemake --rerun-incomplete --latency-wait 300 --snakefile /scratch/tgeorgom/mamba/snakePipes/lib/python3.12/site-packages/snakePipes/workflows/DNA-mapping/Snakefile --jobs 50 --directory /scratch/tgeorgom/CH02/mm10_gencodeM19 --configfile /scratch/tgeorgom/CH02/mm10_gencodeM19/DNA-mapping.config.yaml --keep-going --use-conda --conda-prefix /scratch/tgeorgom/mamba/snakePipes/env --printshellcmds --cluster-config /scratch/tgeorgom/CH02/mm10_gencodeM19/DNA-mapping.cluster_config.yaml --cluster 'sbatch --nodes 1 -p smp-rh7 -A tgeorgom --mem-per-cpu {cluster.memory} -c {threads} -e cluster_logs/{rule}.%j.err -o cluster_logs/{rule}.%j.out -J {rule} '

PID: 13632

---- This analysis has been done using snakePipes version 2.8.0 ----
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cluster nodes: 50
Job stats:
job        count    min threads    max threads
-------  -------  -------------  -------------
all            1              1              1
multiQC        1              1              1
total          2              1              1

Select jobs to execute...

[Wed Jul 24 19:52:26 2024]
rule multiQC:
    input: FastQC_trimmed/...
trimmed/...
Bowtie2/...
Sambamba/...
 deepTools_qc/...
      output: multiQC/multiqc_report.html
    log: multiQC/multiQC.out, multiQC/multiQC.err
    jobid: 259
    reason: Missing output files: multiQC/multiqc_report.html
    resources: mem_mb=9027, disk_mb=9027, tmpdir=<TBD>

multiqc -o multiQC -f  FastQC_trimmed FASTQ_fastp  Sambamba  Bowtie2  deepTools_qc  > multiQC/multiQC.out 2> multiQC/multiQC.err
Submitted job 259 with external jobid 'Submitted batch job 19109829'.
[Wed Jul 24 19:53:55 2024]
Error in rule multiQC:
    jobid: 259
    input: FastQC_trimmed/....
, deepTools_qc/bamPEFragmentSize/fragmentSize.metric.tsv,
Bowtie2.... Sambamba/......
    output: multiQC/multiqc_report.html
    log: multiQC/multiQC.out, multiQC/multiQC.err (check log file(s) for error message)
    conda-env: /scratch/tgeorgom/mamba/snakePipes/env/4f92e0b08e19017ee86e3b4ad470b20b
    shell:
        multiqc -o multiQC -f  FastQC_trimmed FASTQ_fastp  Sambamba  Bowtie2  deepTools_qc  > multiQC/multiQC.out 2> multiQC/multiQC.err
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
    cluster_jobid: Submitted batch job 19109829

Error executing rule multiQC on cluster (jobid: 259, external: Submitted batch job 19109829, jobscript: /scratch/tgeorgom/CH02/mm10_gencodeM19/.snakemake/tmp.ya0pgvlt/snakejob.multiQC.259.sh). For error details see the cluster log and the log files of the involved rule(s).
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-07-24T195157.982252.snakemake.log
--- Config ---------------------------------------------------------------------
config file: /scratch/tgeorgom/CH02/mm10_gencodeM19/DNA-mapping.config.yaml
GCBias: True
NMaskedIndex:
SNPfile:
UMIBarcode: False
UMIDedup: False
UMIDedupOpts: None
UMIDedupSep: _
VCFfile:
aligner: Bowtie2
alignerOpts: None
allele_mode: None
baseDir: /scratch/tgeorgom/mamba/snakePipes/lib/python3.12/site-packages/snakePipes
bcPattern: NNNNCCCCCCCC
bwBinSize: 25
clusterConfig: /home/tgeorgom/shared/cluster.yaml
clusterConfigFile: None
condaEnvDir: /scratch/tgeorgom/mamba/snakePipes/env
configFile: None
configMode: manual
createDAG: False
cutntag: False
dedup: True
downsample: None
emailAddress: None
emailSender: None
ext: .fastq.gz
fastqc: True
fragmentLength: 200
genome: mm10_gencodeM19_spikesTEST
indir: /scratch/tgeorgom/bastet2.ccg.uni-koeln.de/downloads/NGS_CHN02_cnikopoulou_A006200402
insertSizeMax: 5000
keepTemp: False
local: False
mapq: 2
mateOrientation: --fr
maxJobs: 50
max_thread: 8
mode: mapping
oldConfig: None
onlySSL: False
organismsDir: /home/tgeorgom/shared/organisms
outdir: /scratch/tgeorgom/CH02/mm10_gencodeM19
pipeline: dna-mapping
plotFormat: pdf
properPairs: False
qualimap: False
reads: ['_R1_001', '_R2_001']
smtpPassword: None
smtpPort: 0
smtpServer: None
smtpUsername: None
snakemakeOptions:  --rerun-incomplete
strains:
tempDir: /scratch/tgeorgom/temp/
toolsVersion: True
trim: True
trimmer: fastp
trimmerOptions: --trim_poly_g --trim_poly_x -Q -L --correction
verbose: True
workingdir: /scratch/tgeorgom/CH02/mm10_gencodeM19
--------------------------------------------------------------------------------
--- Genome ---------------------------------------------------------------------
config file: /home/tgeorgom/shared/organisms/mm10_gencodeM19_spikesTEST.yaml
blacklist_bed: /home/tgeorgom/assemblies/mm10_gencodeM19_spikesTEST/annotation/blacklist.bed
bowtie2_index: /home/tgeorgom/assemblies/mm10_gencodeM19_spikesTEST/BowtieIndex/genome
bwa_index:
bwa_mem2_index:
bwameth2_index:
bwameth_index:
extended_coding_regions_gtf: /home/tgeorgom/assemblies/mm10_gencodeM19_spikesTEST/annotation/genes.slop.gtf
genes_bed: /home/tgeorgom/assemblies/mm10_gencodeM19_spikesTEST/annotation/genes.bed
genes_gtf: /home/tgeorgom/assemblies/mm10_gencodeM19_spikesTEST/annotation/genes.gtf
genome_2bit: /home/tgeorgom/assemblies/mm10_gencodeM19_spikesTEST/genome_fasta/genome.2bit
genome_dict: /home/tgeorgom/assemblies/mm10_gencodeM19_spikesTEST/genome_fasta/genome.dict
genome_fasta: /home/tgeorgom/assemblies/mm10_gencodeM19_spikesTEST/genome_fasta/genome.fa
genome_index: /home/tgeorgom/assemblies/mm10_gencodeM19_spikesTEST/genome_fasta/genome.fa.fai
genome_size: 2652783500
hisat2_index:
ignoreForNormalization: chrX chrY chrM 
known_splicesites:
rmsk_file:
spikein_blacklist_bed:
spikein_genes_gtf: /home/tgeorgom/assemblies/mm10_gencodeM19_spikesTEST/annotation/spikein_genes.gtf
star_index:
--------------------------------------------------------------------------------

--- Workflow parameters --------------------------------------------------------
samples: [....]
paired: True
read extension: ['_R1_001', '_R2_001']
fastq dir: FASTQ_fastp
maximum insert size (Bowtie2 -X): 5000
--------------------------------------------------------------------------------

--- Environment ----------------------------------------------------------------
$TMPDIR:  /scratch/tgeorgom/temp/
$HOSTNAME:  cheops1
--------------------------------------------------------------------------------

 !!! ERROR in DNA mapping workflow! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Error: snakemake returned an error code of 1, so processing is incomplete!

[katsikora]

Hi,

would you have a corresponding .err log from cluster_logs ?

Best wishes,

Katarzyna

[LeilyR]

I faced the same issue. Using multiqc1.23 solved my issue. Unfortunately dont have the error detail anymore, but it simply couldnt detect any of generated text files as a multiqc compatible file.

[katsikora]

Alright, so an env update is needed :+1:

[sunta3iouxos]

@katsikora if I am not mistaken multiqc in order to work properly needs a different conda environment than some of the programs used in a standard bioinformatics analysis. I do not know if this was an issue with a specific version. I solved this issue by isolating multiqc from a main conda environment.