HPC Execution problem when changing to V.1.1.0

[luisalbertoc95]

Hi!

We used to run V1.0.5 with no big issues , but after start using the new release (V.1.1.0) we are having problems with the preprocessing rules (remove_5prime_primer). The same dataset ran successfully with the previous version.

We are using RoundAb preprocessing option. Because of the final line of the report, we think there is a conflict between Snakemake and slurm but we are not sure of what could be happening.

Furthermore, the file hecatomb.crashreport.log is empty after the execution failure.

I have attached the Snakemake report.

Thank you!

--

rule remove_5prime_primer: input: /scratch/sahlab/Luis/FGT_Virome/00_fastq_sahlab/I3278_R1.fastq.gz, /scratch/sahlab/Luis/FGT_Virome/00_fastq_sahlab/I3278_R2.fastq.gz, /ref/sahlab /software/anaconda3/envs/hecatomb_v1.1.0/snakemake/workflow/../../databases/contaminants/primerB.fa output: hecatomb.out/processing/temp/p01/I3278_R1.s1.out.fastq, hecatomb.out/processing/temp/p01/I3278_R2.s1.out.fastq, hecatomb.out/processing/stats/p01/I3278.s1.stats.tsv log: hecatomb.out/stderr/remove_5prime_primer.I3278.log jobid: 342 benchmark: hecatomb.out/benchmarks/remove_5prime_primer.I3278.txt reason: Missing output files: hecatomb.out/processing/temp/p01/I3278_R1.s1.out.fastq, hecatomb.out/processing/temp/p01/I3278_R2.s1.out.fastq wildcards: sample=I3278 threads: 8 resources: mem_mb=16000, disk_mb=1000, tmpdir=, time=1440, javaAlloc=14400 Error executing rule remove_5prime_primer on cluster (jobid: 980, external: Submitted batch job 2465688, jobscript: /scratch/sahlab/Luis/FGT_Virome/02_sahlabSamples_HecatombRun_TestNewHec/.snakemake/tmp.jys3bc5q/snakejob.remove_5prime_primer.980.sh). For error details see the cluster log and the log files of the involved rule(s).

    bbduk.sh in=/scratch/sahlab/Luis/FGT_Virome/00_fastq_sahlab/I3278_R1.fastq.gz in2=/scratch/sahlab/Luis/FGT_Virome/00_fastq_sahlab/I3278_R2.fastq.gz             ref=/ref/sahlab/software/anaconda3/envs/hecatomb_v1.1.0/snakemake/workflow/../../databases/contaminants/primerB.fa             out=hecatomb.out/processing/temp/p01/I3278_R1.s1.out.fastq out2=hecatomb.out/processing/temp/p01/I3278_R2.s1.out.fastq             stats=hecatomb.out/processing/stats/p01/I3278.s1.stats.tsv             k=16 hdist=1 mink=11 ktrim=l restrictleft=20             removeifeitherbad=f trimpolya=10 ordered=t rcomp=f ow=t             threads=8 -Xmx16000m 2> hecatomb.out/stderr/remove_5prime_primer.I3278.log
    rm hecatomb.out/stderr/remove_5prime_primer.I3278.log

sacct: fatal: Bad job/step specified: Submitted

[beardymcjohnface]

Hi Luis, I'm not sure what the problem might be here other than an issue I had a while ago after updating Snakemake. Are you using a profile to run on the cluster? if so can you show me your profile config?

[luisalbertoc95]

HI Mike,

Here the profile config file.

Thank you,

Luis config.txt

[beardymcjohnface]

Hi Luis, Yes I see the problem. I've run into this before but forgot to update the example profile with Hecatomb. You need to add --parsable to the sbatch command following a Snakemake update. Try this:

cluster:
  mkdir -p logs/{rule}/ &&
  sbatch
    --cpus-per-task={threads}
    --mem={resources.mem_mb}
    --time={resources.time}
    --job-name=smk-{rule}
    --parsable
    --output=logs/{rule}/{jobid}.out
    --error=logs/{rule}/{jobid}.err
default-resources:
  - mem_mb=2000
  - time=1440
jobs: 100
latency-wait: 50
local-cores: 8
restart-times: 1
max-jobs-per-second: 20
keep-going: True
rerun-incomplete: True
printshellcmds: True
scheduler: greedy
use-conda: True
conda-frontend: mamba
cluster-status: ~/.config/snakemake/slurm/slurm-status.py
max-status-checks-per-second: 10

[luisalbertoc95]

Now it's working!

Thanks a lot Mike.

Luis