The Nextpolish polishing of a barley assembly generated using R10.4 ONT long-reads using ~x40 illumina 150 bp PE reads stops after approximately 1h due to a out-of-memory and time limit issue and i cannot seem to fix it. Hope that you can help!

SLURM resources:

SBATCH --job-name=polishing_nextpolish

SBATCH --output=./out/polishing_nextpolish.out

SBATCH --error=./err/polishing_nextpolish.err

SBATCH --ntasks=30

SBATCH --cpus-per-task=1

SBATCH --mem=600G

SBATCH --time=48:00:00

run.cfg: [General] job_type = slurm job_prefix = nextPolish task = best rewrite = yes rerun = 3 parallel_jobs = 6 multithread_jobs = 5 genome = /projects/mjolnir1/people/bgn602/steps/01_comp_study/flye/parameter_eval/t2/A3_em2_1/A3_em2_1.fasta genome_size = auto workdir = /projects/mjolnir1/people/bgn602/steps/01_comp_study/nextpolish/parameter_eval/t2/A3_em2_1 polish_options = -p {multithread_jobs}

[sgs_option] sgs_fofn = /projects/mjolnir1/people/bgn602/steps/01_comp_study/nextpolish/parameter_eval/t2/A3_em2_1/sgs.fofn sgs_options = -max_depth 100 -bwa

sgs.fofn /projects/mjolnir1/people/bgn602/steps/01_comp_study/fastp/illu/hv_em2_1_illu_processed.fastq /projects/mjolnir1/people/bgn602/steps/01_comp_study/fastp/illu/hv_em2_2_illu_processed.fastq

info file: [2302180 INFO] 2023-11-17 15:59:42 NextPolish start... [2302180 INFO] 2023-11-17 15:59:42 version:v1.4.1 logfile:pid2302180.log.info [2302180 WARNING] 2023-11-17 15:59:42 Re-write workdir [2302180 WARNING] 2023-11-17 15:59:46 Delete task: 5 due to missing lgs_fofn. [2302180 WARNING] 2023-11-17 15:59:46 Delete task: 5 due to missing lgs_fofn. [2302180 WARNING] 2023-11-17 15:59:46 Delete task: 6 due to missing hifi_fofn. [2302180 WARNING] 2023-11-17 15:59:46 Delete task: 6 due to missing hifi_fofn. [2302180 INFO] 2023-11-17 15:59:46 scheduled tasks: [1, 2, 1, 2] [2302180 INFO] 2023-11-17 15:59:46 options: [2302180 INFO] 2023-11-17 15:59:46 rerun: 3 rewrite: 1 kill: None cleantmp: 0 use_drmaa: 0 submit: None deltmp: False job_type: slurm sgs_unpaired: 0 sgs_rm_nread: 1 lgs_read_type: parallel_jobs: 6 align_threads: 5 check_alive: None sgs_max_depth: 50 task: [1, 2, 1, 2] job_id_regex: None lgs_max_depth: 100 multithread_jobs: 5 lgs_max_read_len: 0 hifi_max_depth: 100 polish_options: -p 5 lgs_block_size: 500M lgs_min_read_len: 1k hifi_max_read_len: 0 hifi_block_size: 500M hifi_min_read_len: 1k job_prefix: nextPolish genome_size: 3900762858 sgs_block_size: 500000000 sgs_use_duplicate_reads: 0 lgs_minimap2_options: -x map-ont hifi_minimap2_options: -x map-pb sgs_align_options: bwa mem -p -t 5 workdir: /projects/mjolnir1/people/bgn602/steps/01_comp_study/nextpolish/parameter_eval/t2/A3_em2_1 genome: /projects/mjolnir1/people/bgn602/steps/01_comp_study/flye/parameter_eval/t2/A3_em2_1/A3_em2_1.fasta sgs_fofn: /projects/mjolnir1/people/bgn602/steps/01_comp_study/nextpolish/parameter_eval/t2/A3_em2_1/sgs.fofn snp_phase: /projects/mjolnir1/people/bgn602/steps/01_comp_study/nextpolish/parameter_eval/t2/A3_em2_1/%02d.snp_phase snp_valid: /projects/mjolnir1/people/bgn602/steps/01_comp_study/nextpolish/parameter_eval/t2/A3_em2_1/%02d.snp_valid lgs_polish: /projects/mjolnir1/people/bgn602/steps/01_comp_study/nextpolish/parameter_eval/t2/A3_em2_1/%02d.lgs_polish kmer_count: /projects/mjolnir1/people/bgn602/steps/01_comp_study/nextpolish/parameter_eval/t2/A3_em2_1/%02d.kmer_count hifi_polish: /projects/mjolnir1/people/bgn602/steps/01_comp_study/nextpolish/parameter_eval/t2/A3_em2_1/%02d.hifi_polish score_chain: /projects/mjolnir1/people/bgn602/steps/01_comp_study/nextpolish/parameter_eval/t2/A3_em2_1/%02d.score_chain [2302180 INFO] 2023-11-17 15:59:46 step 0 and task 1 start: [2302180 INFO] 2023-11-17 15:59:51 Total jobs: 3 [2302180 INFO] 2023-11-17 15:59:51 Submitted jobID:[7449925] jobCmd:[/projects/mjolnir1/people/bgn602/steps/01_comp_study/nextpolish/parameter_eval/t2/A3_em2_1/00.score_chain/01.db_split.sh.work/db_split1/nextPolish.sh] in the slurm_cycle. [2302180 INFO] 2023-11-17 15:59:51 Submitted jobID:[7449926] jobCmd:[/projects/mjolnir1/people/bgn602/steps/01_comp_study/nextpolish/parameter_eval/t2/A3_em2_1/00.score_chain/01.db_split.sh.work/db_split2/nextPolish.sh] in the slurm_cycle. [2302180 INFO] 2023-11-17 15:59:51 Submitted jobID:[7449927] jobCmd:[/projects/mjolnir1/people/bgn602/steps/01_comp_study/nextpolish/parameter_eval/t2/A3_em2_1/00.score_chain/01.db_split.sh.work/db_split3/nextPolish.sh] in the slurm_cycle. [2302180 ERROR] 2023-11-17 17:00:34 db_split failed: please check the following logs: [2302180 ERROR] 2023-11-17 17:00:34 /projects/mjolnir1/people/bgn602/steps/01_comp_study/nextpolish/parameter_eval/t2/A3_em2_1/00.score_chain/01.db_split.sh.work/db_split1/nextPolish.sh.e [2302180 ERROR] 2023-11-17 17:00:34 /projects/mjolnir1/people/bgn602/steps/01_comp_study/nextpolish/parameter_eval/t2/A3_em2_1/00.score_chain/01.db_split.sh.work/db_split2/nextPolish.sh.e

db_split1/nextPolish.sh.e: hostname

hostname cd /projects/mjolnir1/people/bgn602/steps/01_comp_study/nextpolish/parameter_eval/t2/A3_em2_1/00.score_chain/01.db_split.sh.work/db_split1
cd /projects/mjolnir1/people/bgn602/steps/01_comp_study/nextpolish/parameter_eval/t2/A3_em2_1/00.score_chain/01.db_split.sh.work/db_split1 ( time /maps/projects/mjolnir1/people/bgn602/packages/nextpolish/bin/seq_split -d /projects/mjolnir1/people/bgn602/steps/01_comp_study/nextpolish/parameter_eval/t2/A3_em2_1 -m 500000000 -n 3 -t 3 -i 1 -s 390076285800 -p input.sgspart /projects/mjolnir1/people/bgn602/steps/01_comp_study/nextpolish/parameter_eval/t2/A3_em2_1/sgs.fofn )
/maps/projects/mjolnir1/people/bgn602/packages/nextpolish/bin/seq_split -d /projects/mjolnir1/people/bgn602/steps/01_comp_study/nextpolish/parameter_eval/t2/A3_em2_1 -m 500000000 -n 3 -t 3 -i 1 -s 390076285800 -p input.sgspart /projects/mjolnir1/people/bgn602/steps/01_comp_study/nextpolish/parameter_eval/t2/A3_em2_1/sgs.fofn slurmstepd: error: JOB 7750998 ON mjolnircomp05fl CANCELLED AT 2023-11-24T03:54:28 DUE TO TIME LIMIT

db_split2/nextPolish.sh.e: hostname

hostname cd /projects/mjolnir1/people/bgn602/steps/01_comp_study/nextpolish/parameter_eval/t2/A3_em2_1/00.score_chain/01.db_split.sh.work/db_split2
cd /projects/mjolnir1/people/bgn602/steps/01_comp_study/nextpolish/parameter_eval/t2/A3_em2_1/00.score_chain/01.db_split.sh.work/db_split2 ( time /maps/projects/mjolnir1/people/bgn602/packages/nextpolish/bin/bwa index -p /projects/mjolnir1/people/bgn602/steps/01_comp_study/nextpolish/parameter_eval/t2/A3_em2_1/00.score_chain/input.genome.fasta.sgs /projects/mjolnir1/people/bgn602/steps/01_comp_study/nextpolish/parameter_eval/t2/A3_em2_1/00.score_chain/input.genome.fasta )
/maps/projects/mjolnir1/people/bgn602/packages/nextpolish/bin/bwa index -p /projects/mjolnir1/people/bgn602/steps/01_comp_study/nextpolish/parameter_eval/t2/A3_em2_1/00.score_chain/input.genome.fasta.sgs /projects/mjolnir1/people/bgn602/steps/01_comp_study/nextpolish/parameter_eval/t2/A3_em2_1/00.score_chain/input.genome.fasta [bwa_index] Pack FASTA... 33.42 sec [bwa_index] Construct BWT for the packed sequence... [BWTIncCreate] textLength=7801525716, availableWord=560944144 [BWTIncConstructFromPacked] 10 iterations done. 99999988 characters processed. [BWTIncConstructFromPacked] 20 iterations done. 199999988 characters processed. [BWTIncConstructFromPacked] 30 iterations done. 299999988 characters processed. [BWTIncConstructFromPacked] 40 iterations done. 399999988 characters processed. [BWTIncConstructFromPacked] 50 iterations done. 499999988 characters processed. [BWTIncConstructFromPacked] 60 iterations done. 599999988 characters processed. [BWTIncConstructFromPacked] 70 iterations done. 699999988 characters processed. [BWTIncConstructFromPacked] 80 iterations done. 799999988 characters processed. [BWTIncConstructFromPacked] 90 iterations done. 899999988 characters processed. [BWTIncConstructFromPacked] 100 iterations done. 999999988 characters processed. [BWTIncConstructFromPacked] 110 iterations done. 1099999988 characters processed. [BWTIncConstructFromPacked] 120 iterations done. 1199999988 characters processed. [BWTIncConstructFromPacked] 130 iterations done. 1299999988 characters processed. [BWTIncConstructFromPacked] 140 iterations done. 1399999988 characters processed. [BWTIncConstructFromPacked] 150 iterations done. 1499999988 characters processed. [BWTIncConstructFromPacked] 160 iterations done. 1599999988 characters processed. [BWTIncConstructFromPacked] 170 iterations done. 1699999988 characters processed. [BWTIncConstructFromPacked] 180 iterations done. 1799999988 characters processed. [BWTIncConstructFromPacked] 190 iterations done. 1899999988 characters processed. [BWTIncConstructFromPacked] 200 iterations done. 1999999988 characters processed. [BWTIncConstructFromPacked] 210 iterations done. 2099999988 characters processed. [BWTIncConstructFromPacked] 220 iterations done. 2199999988 characters processed. [BWTIncConstructFromPacked] 230 iterations done. 2299999988 characters processed. [BWTIncConstructFromPacked] 240 iterations done. 2399999988 characters processed. [BWTIncConstructFromPacked] 250 iterations done. 2499999988 characters processed. [BWTIncConstructFromPacked] 260 iterations done. 2599999988 characters processed. [BWTIncConstructFromPacked] 270 iterations done. 2699999988 characters processed. [BWTIncConstructFromPacked] 280 iterations done. 2799999988 characters processed. [BWTIncConstructFromPacked] 290 iterations done. 2899999988 characters processed. [BWTIncConstructFromPacked] 300 iterations done. 2999999988 characters processed. [BWTIncConstructFromPacked] 310 iterations done. 3099999988 characters processed. [BWTIncConstructFromPacked] 320 iterations done. 3199999988 characters processed. [BWTIncConstructFromPacked] 330 iterations done. 3299999988 characters processed. [BWTIncConstructFromPacked] 340 iterations done. 3399999988 characters processed. [BWTIncConstructFromPacked] 350 iterations done. 3499999988 characters processed. [BWTIncConstructFromPacked] 360 iterations done. 3599999988 characters processed. [BWTIncConstructFromPacked] 370 iterations done. 3699999988 characters processed. [BWTIncConstructFromPacked] 380 iterations done. 3799999988 characters processed. [BWTIncConstructFromPacked] 390 iterations done. 3899999988 characters processed. [BWTIncConstructFromPacked] 400 iterations done. 3999999988 characters processed. [BWTIncConstructFromPacked] 410 iterations done. 4099999988 characters processed. [BWTIncConstructFromPacked] 420 iterations done. 4199999988 characters processed. [BWTIncConstructFromPacked] 430 iterations done. 4299999988 characters processed. [BWTIncConstructFromPacked] 440 iterations done. 4399999988 characters processed. [BWTIncConstructFromPacked] 450 iterations done. 4499999988 characters processed. [BWTIncConstructFromPacked] 460 iterations done. 4599999988 characters processed. [BWTIncConstructFromPacked] 470 iterations done. 4699999988 characters processed. [BWTIncConstructFromPacked] 480 iterations done. 4799999988 characters processed. [BWTIncConstructFromPacked] 490 iterations done. 4899999988 characters processed. [BWTIncConstructFromPacked] 500 iterations done. 4999999988 characters processed. [BWTIncConstructFromPacked] 510 iterations done. 5099999988 characters processed. [BWTIncConstructFromPacked] 520 iterations done. 5199999988 characters processed. [BWTIncConstructFromPacked] 530 iterations done. 5299999988 characters processed. [BWTIncConstructFromPacked] 540 iterations done. 5399999988 characters processed. [BWTIncConstructFromPacked] 550 iterations done. 5499999988 characters processed. [BWTIncConstructFromPacked] 560 iterations done. 5599999988 characters processed. [BWTIncConstructFromPacked] 570 iterations done. 5699999988 characters processed. [BWTIncConstructFromPacked] 580 iterations done. 5799999988 characters processed. [BWTIncConstructFromPacked] 590 iterations done. 5899999988 characters processed. [BWTIncConstructFromPacked] 600 iterations done. 5999999988 characters processed. [BWTIncConstructFromPacked] 610 iterations done. 6099999988 characters processed. [BWTIncConstructFromPacked] 620 iterations done. 6199999988 characters processed. [BWTIncConstructFromPacked] 630 iterations done. 6299999988 characters processed. [BWTIncConstructFromPacked] 640 iterations done. 6399999988 characters processed. [BWTIncConstructFromPacked] 650 iterations done. 6499999988 characters processed. [BWTIncConstructFromPacked] 660 iterations done. 6599999988 characters processed. [BWTIncConstructFromPacked] 670 iterations done. 6699999988 characters processed. [BWTIncConstructFromPacked] 680 iterations done. 6799999988 characters processed. [BWTIncConstructFromPacked] 690 iterations done. 6899999988 characters processed. [BWTIncConstructFromPacked] 700 iterations done. 6999999988 characters processed. [BWTIncConstructFromPacked] 710 iterations done. 7099999988 characters processed. /var/spool/slurm//job7750999/slurm_script: line 5: 1590822 Killed /maps/projects/mjolnir1/people/bgn602/packages/nextpolish/bin/bwa index -p /projects/mjolnir1/people/bgn602/steps/01_comp_study/nextpolish/parameter_eval/t2/A3_em2_1/00.score_chain/input.genome.fasta.sgs /projects/mjolnir1/people/bgn602/steps/01_comp_study/nextpolish/para$

real 50m38.542s user 49m35.333s sys 0m42.718s slurmstepd: error: Detected 1 oom-kill event(s) in StepId=7750999.batch cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler.

paralleltask/cluster.cfg [SGE] submit = qsub -pe smp {cpu} -l vf={mem} -o {out} -e {err} {script} kill = qdel {job_id} check-alive = qstat -j {job_id} job-id-regex = (\d+)

[PBS/TORQUE] submit = qsub -l nodes=1:ppn={cpu},mem={mem} -o {out} -e {err} {script} kill = qdel {job_id} check-alive = qstat {job_id} job-id-regex = (\d+)

[LSF] submit = bsub -n {cpu} -R rusage[mem={mem}] -o {out} -e {err} {script} kill = bkill {job_id} check-alive = bjobs {job_id} job-id-regex = Job <(\d+)>

[SLURM] submit = sbatch --cpus-per-task=1 --mem-per-cpu=20G --time 48:00:00 -o {out} -e {err} {script} kill = scancel {job_id} check-alive = squeue -j {job_id} job-id-regex = Submitted batch job (\d+)

Nextomics / NextPolish

SLURM out of memory and time... #125