johannanvs
commented
1 year ago Update: when I convert it into a sbatch script it completes in less than 10 minutes. I still use the same singularity image (see rule converted to sbatch script below). Not sure what is going on here...
#!/bin/bash -l
#SBATCH -A compute_id
#SBATCH -p core
#SBATCH -n 5
#SBATCH -t 04:00:00
#SBATCH -J multiqc_historical_raw
# Summarize all fastqc results for all trimmed and merged historical samples.
######################
## Script variables ##
######################
# General variables
generode=/path/to/GenErode/workflow
indir="${generode}/data/raw_reads_symlinks/historical/stats/"
outdir="${generode}/data/raw_reads_symlinks/historical/stats/multiqc/"
log="${generode}/data/logs/1.1_fastq_processing/historical/multiqc_historical_raw.log"
singularity_image="docker://quay.io/biocontainers/multiqc:1.9--pyh9f0ad1d_0"
###################
## Echo job info ##
###################
echo ""
echo "*******************************************************
echo " JOB INFO: "
echo "------------------------------------------------------"
echo -e "Date:"
date
echo -e "Input directory:" ${indir}
echo -e "Output directory:" ${outdir}
echo ""
echo -e "Slurm job ID:" $SLURM_JOB_ID
echo -e "Slurm job name:" $SLURM_JOB_NAME
echo "*******************************************************
echo ""
###########
## Shell ##
###########
# Run MultiQC
echo ""
echo ">>> multiQC <<<"
echo ""
singularity run ${singularity_image} \
multiqc -f ${indir} -o ${outdir} 2> ${log}
###########
###########
echo ""
echo "-------------------------------------------------------------------------"
echo "Job finished by:"
date
echo "-------------------------------------------------------------------------"
echo ""
echo ""
johnne
verku
Whenever I try to run multiQC rules, the rule gets stuck on
[INFO ] multiqc : Searching : /path/to/stats/, and my job gets cancelled due to reaching the requested job time limit. I've tried increasing the time to 6 hours instead of 2, but all this time it's still searching for files. For rule multiqc_historical_raw I've got around 550 files, but still, should it really take that long to search?log file:
slurm standard error file: