Samtools sort out-of-memory

[RenscoHogers]

Describe the bug While running NextDenovo on SLURM, I run into a samtools sort error, seemingly due to too little reserved memory.

Error message Main task: [1700406 INFO] 2023-05-25 08:28:29 Submitted jobID:[42186471] jobCmd:[/home/WUR/hoger006/lustre_dir/Data/Dutch_Belted/NextDenovo_out/03.ctg_graph/02.ctg_align.sh.work/ctg_align16/nextDenovo.sh] in the slurm_cycle. [1700406 ERROR] 2023-05-25 08:55:12 ESC[35mctg_align failed: please check the following logs:ESC[0m [1700406 ERROR] 2023-05-25 08:55:12 ESC[35m/home/WUR/hoger006/lustre_dir/Data/Dutch_Belted/NextDenovo_out/03.ctg_graph/02.ctg_align.sh.work/ctg_align01/nextDenovo.sh.eESC[0m

Subtask: hostname

• hostname cd /home/WUR/hoger006/lustre_dir/Data/Dutch_Belted/NextDenovo_out/03.ctg_graph/02.ctg_align.sh.work/ctg_align01
• cd /home/WUR/hoger006/lustre_dir/Data/Dutch_Belted/NextDenovo_out/03.ctg_graph/02.ctg_align.sh.work/ctg_align01 time /lustre/nobackup/WUR/ABGC/hoger006/Tools/mambaforge/envs/nextdenovo/lib/python3.10/site-packages/nextdenovo/bin/minimap2-nd --step 3 -x map-ont -a -t 32 /home/WUR/hoger006/lustre_dir/Data/Dutch_Belted/NextDenovo_out/03.ctg_graph/01.ctg_graph.sh.work/ctg_graph1/nd.asm.p.fasta /home/WUR/hoger006/lustre_dir/Data/Dutch_Belted/NextDenovo_out/01.raw_align/input.seed.003.2bit|/lustre/nobackup/WUR/ABGC/hoger006/Tools/mambaforge/envs/nextdenovo/lib/python3.10/site-packages/nextdenovo/bin/bam_sort -i -@ 32 -o input.seed.003.2bit.sort.bam
• • time /lustre/nobackup/WUR/ABGC/hoger006/Tools/mambaforge/envs/nextdenovo/lib/python3.10/site-packages/nextdenovo/bin/minimap2-nd/lustre/nobackup/WUR/ABGC/hoger006/Tools/mambaforge/envs/nextdenovo/lib/python3.10/site-packages/nextdenovo/bin/bam_sort --step -i 3 -@ -x 32 map-ont -o -a input.seed.003.2bit.sort.bam -t 32 /home/WUR/hoger006/lustre_dir/Data/Dutch_Belted/NextDenovo_out/03.ctg_graph/01.ctg_graph.sh.work/ctg_graph1/nd.asm.p.fasta /home/WUR/hoger006/lustre_dir/Data/Dutch_Belted/NextDenovo_out/01.raw_align/input.seed.003.2bit [M::mm_idx_gen::58.8211.66] collected minimizers [M::mm_idx_gen::60.6662.35] sorted minimizers [M::main::60.6692.35] loaded/built the index for 3033 target sequence(s) [M::mm_mapopt_update::62.5772.31] mid_occ = 660 [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 3033 [M::mm_idx_stat::63.6932.29] distinct minimizers: 94498812 (41.66% are singletons); average occurrences: 4.866; average spacing: 5.331 [M::worker_pipeline::158.26617.91] mapped 22594 sequences [M::worker_pipeline::239.393*22.00] mapped 22805 sequences Command terminated by signal 9 5927.23user 178.00system 4:27.79elapsed 2279%CPU (0avgtext+0avgdata 33847516maxresident)k 5931360inputs+24outputs (0major+85737664minor)pagefaults 0swaps [W::sam_read1] Parse error at line 113646 samtools sort: truncated file. Aborting slurmstepd: error: Detected 1 oom-kill event(s) in StepId=42185229.batch. Some of your processes may have been killed by the cgroup out-of-memory handler.

Config file [General] job_type = slurm submit = auto job_prefix = nextDenovo task = all rewrite = yes deltmp = yes parallel_jobs = 32 input_type = raw read_type = ont input_fofn = /home/WUR/hoger006/lustre_dir/Data/Dutch_Belted/NextDenovo_out/input.fofn workdir = /home/WUR/hoger006/lustre_dir/Data/Dutch_Belted/NextDenovo_out

[correct_option] read_cutoff = 1K genome_size = 827922386 sort_options = -m 100g -t 32 minimap2_options_raw = -t 32 -m 200g pa_correction = 8 correction_options = -p 32

[assemble_option] minimap2_options_cns = -t 32 nextgraph_options = -a 1

Operating system Which operating system and version are you using? Ubuntu 20.04 LTS

GCC What version of GCC are you using? 11.2.0

Python What version of Python are you using? 3.10.8

NextDenovo What version of NextDenovo are you using? 2.5.2

[moold]

see here to set submit. such as sbatch --cpus-per-task={cpu} --mem-per-cpu=2g -o {out} -e {err} {script}

or set submit = sbatch --cpus-per-task={cpu} --mem-per-cpu=2g -o {out} -e {err} {script} in your config file, see details

[DaniPaulo]

Hi @RenscoHogers. I'm still trying to figure out how to run NextDenovo in a HPC environment using SLURM. Would you be able to share your script.slurm.sh with me?