search

Nextomics / NextPolish

Fast and accurately polish the genome generated by long reads.
GNU General Public License v3.0
205 stars 28 forks source link

polish_genome failed #128

Open liuyca1 opened 6 months ago

liuyca1 commented 6 months ago

Describe the bug When I used second-generation sequencing data (MGI_DNBSEQ platform) to optimize Nanopore assembly (using nextdenovo software) data, an error occurred. The running log is as follows:

run.cfg [General] job_type = local job_prefix = nextPolish task = best rewrite = no deltmp = yes rerun = 3 parallel_jobs = 3 multithread_jobs = 10 genome = /public/home/user/Plant_genome/assembly/nextDenovo_output/03.ctg_graph/nd.asm.fasta genome_size = 800Mb workdir = ./01_rundir polish_options = -p {multithread_jobs}

[sgs_option] sgs_fofn = /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/sgs.fofn sgs_options = -max_depth 100 -bwa

Error message $cat log [302167 INFO] 2024-03-15 09:01:07 NextPolish start... [302167 INFO] 2024-03-15 09:01:07 version:1.4.1 logfile:pid302167.log.info [302167 WARNING] 2024-03-15 09:01:07 Delete task: 5 due to missing lgs_fofn. [302167 WARNING] 2024-03-15 09:01:07 Delete task: 5 due to missing lgs_fofn. [302167 WARNING] 2024-03-15 09:01:07 Delete task: 6 due to missing hifi_fofn. [302167 WARNING] 2024-03-15 09:01:07 Delete task: 6 due to missing hifi_fofn. [302167 INFO] 2024-03-15 09:01:07 scheduled tasks: [1, 2, 1, 2] [302167 INFO] 2024-03-15 09:01:07 options: [302167 INFO] 2024-03-15 09:01:07 rerun: 3 rewrite: 0 kill: None cleantmp: 0 use_drmaa: 0 deltmp: True submit: None job_type: local sgs_unpaired: 0 sgs_rm_nread: 1 lgs_read_type:
parallel_jobs: 3 check_alive: None align_threads: 10 task: [1, 2, 1, 2] job_id_regex: None sgs_max_depth: 100 lgs_max_depth: 100 lgs_max_read_len: 0 hifi_max_depth: 100 multithread_jobs: 10 lgs_block_size: 500M lgs_min_read_len: 1k hifi_max_read_len: 0 polish_options: -p 10 hifi_block_size: 500M hifi_min_read_len: 1k job_prefix: nextPolish genome_size: 800000000 sgs_block_size: 500000000 sgs_use_duplicate_reads: 0 lgs_minimap2_options: -x map-ont hifi_minimap2_options: -x map-pb sgs_align_options: bwa mem -p -t 10 sgs_fofn: /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/sgs.fofn workdir: /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/./01_rundir genome: /public/home/user/Plant_genome/assembly/nextDenovo_output/03.ctg_graph/nd.asm.fasta snp_phase: /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/./01_rundir/%02d.snp_phase snp_valid: /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/./01_rundir/%02d.snp_valid lgs_polish: /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/./01_rundir/%02d.lgs_polish kmer_count: /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/./01_rundir/%02d.kmer_count hifi_polish: /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/./01_rundir/%02d.hifi_polish score_chain: /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/./01_rundir/%02d.score_chain [302167 INFO] 2024-03-15 09:01:07 step 0 and task 1 start: [302167 INFO] 2024-03-15 09:01:12 Total jobs: 3 [302167 INFO] 2024-03-15 09:01:12 Submitted jobID:[302172] jobCmd:[/public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/01.db_split.sh.work/db_split1/nextPolish.sh] in the local_cycle. [302167 INFO] 2024-03-15 09:01:13 Submitted jobID:[302188] jobCmd:[/public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/01.db_split.sh.work/db_split2/nextPolish.sh] in the local_cycle. [302167 INFO] 2024-03-15 09:01:13 Submitted jobID:[302193] jobCmd:[/public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/01.db_split.sh.work/db_split3/nextPolish.sh] in the local_cycle. [302167 INFO] 2024-03-15 11:36:27 db_split done [302167 INFO] 2024-03-15 11:36:32 Total jobs: 3 [302167 INFO] 2024-03-15 11:36:32 Submitted jobID:[398779] jobCmd:[/public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/02.map.ref.sh.work/map_genome1/nextPolish.sh] in the local_cycle. [302167 INFO] 2024-03-15 11:36:33 Submitted jobID:[398782] jobCmd:[/public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/02.map.ref.sh.work/map_genome2/nextPolish.sh] in the local_cycle. [302167 INFO] 2024-03-15 11:36:33 Submitted jobID:[398796] jobCmd:[/public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/02.map.ref.sh.work/map_genome3/nextPolish.sh] in the local_cycle. [302167 INFO] 2024-03-15 14:06:27 align_genome done [302167 INFO] 2024-03-15 14:06:33 Total jobs: 1 [302167 INFO] 2024-03-15 14:06:33 Submitted jobID:[47847] jobCmd:[/public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/03.merge.bam.sh.work/merge_bam1/nextPolish.sh] in the local_cycle. [302167 INFO] 2024-03-15 15:03:05 merge_bam done [302167 INFO] 2024-03-15 15:03:05 remove temporary result: /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/02.map.ref.sh.work/map_genome1/sgs.part000.sort.bam [302167 INFO] 2024-03-15 15:03:06 remove temporary result: /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/02.map.ref.sh.work/map_genome2/sgs.part001.sort.bam [302167 INFO] 2024-03-15 15:03:10 remove temporary result: /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/02.map.ref.sh.work/map_genome3/sgs.part002.sort.bam [302167 INFO] 2024-03-15 15:03:16 Total jobs: 3 [302167 INFO] 2024-03-15 15:03:16 Submitted jobID:[83886] jobCmd:[/public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/04.polish.ref.sh.work/polish_genome1/nextPolish.sh] in the local_cycle. [302167 INFO] 2024-03-15 15:03:16 Submitted jobID:[83891] jobCmd:[/public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/04.polish.ref.sh.work/polish_genome2/nextPolish.sh] in the local_cycle. [302167 INFO] 2024-03-15 15:03:17 Submitted jobID:[83897] jobCmd:[/public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/04.polish.ref.sh.work/polish_genome3/nextPolish.sh] in the local_cycle. [83897 CRITICAL] 2024-03-15 15:03:19 Command '/bin/sh /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/04.polish.ref.sh.work/polish_genome3/nextPolish.sh > /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/04.polish.ref.sh.work/polish_genome3/nextPolish.sh.o 2> /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/04.polish.ref.sh.work/polish_genome3/nextPolish.sh.e' returned non-zero exit status 1, error info: . [83886 CRITICAL] 2024-03-15 15:03:19 Command '/bin/sh /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/04.polish.ref.sh.work/polish_genome1/nextPolish.sh > /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/04.polish.ref.sh.work/polish_genome1/nextPolish.sh.o 2> /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/04.polish.ref.sh.work/polish_genome1/nextPolish.sh.e' returned non-zero exit status 1, error info: . [83891 CRITICAL] 2024-03-15 15:03:19 Command '/bin/sh /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/04.polish.ref.sh.work/polish_genome2/nextPolish.sh > /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/04.polish.ref.sh.work/polish_genome2/nextPolish.sh.o 2> /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/04.polish.ref.sh.work/polish_genome2/nextPolish.sh.e' returned non-zero exit status 1, error info: . Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): File "/public/home/user/software/anaconda3/envs/nextpolish/bin/nextPolish", line 553, in File "/public/home/user/software/anaconda3/envs/nextpolish/bin/nextPolish", line 553, in File "/public/home/user/software/anaconda3/envs/nextpolish/bin/nextPolish", line 553, in main(args) main(args) main(args) File "/public/home/user/software/anaconda3/envs/nextpolish/bin/nextPolish", line 502, in main File "/public/home/user/software/anaconda3/envs/nextpolish/bin/nextPolish", line 502, in main File "/public/home/user/software/anaconda3/envs/nextpolish/bin/nextPolish", line 502, in main task.run.start() task.run.start() File "/public/home/user/software/anaconda3/envs/nextpolish/lib/python3.11/site-packages/paralleltask/task_control.py", line 344, in start File "/public/home/user/software/anaconda3/envs/nextpolish/lib/python3.11/site-packages/paralleltask/task_control.py", line 344, in start task.run.start() File "/public/home/user/software/anaconda3/envs/nextpolish/lib/python3.11/site-packages/paralleltask/task_control.py", line 344, in start self._start() self._start() File "/public/home/user/software/anaconda3/envs/nextpolish/lib/python3.11/site-packages/paralleltask/task_control.py", line 368, in _start File "/public/home/user/software/anaconda3/envs/nextpolish/lib/python3.11/site-packages/paralleltask/task_control.py", line 368, in _start self._start() File "/public/home/user/software/anaconda3/envs/nextpolish/lib/python3.11/site-packages/paralleltask/task_control.py", line 368, in _start self.submit(job) File "/public/home/user/software/anaconda3/envs/nextpolish/lib/python3.11/site-packages/paralleltask/task_control.py", line 252, in submit self.submit(job) File "/public/home/user/software/anaconda3/envs/nextpolish/lib/python3.11/site-packages/paralleltask/task_control.py", line 252, in submit self.submit(job) File "/public/home/user/software/anaconda3/envs/nextpolish/lib/python3.11/site-packages/paralleltask/taskcontrol.py", line 252, in submit , stdout, = self.run(job.cmd) , stdout, _ = self.run(job.cmd) ^^^^^^^^^^^^^^^^^ File "/public/home/user/software/anaconda3/envs/nextpolish/lib/python3.11/site-packages/paralleltask/task_control.py", line 288, in run ^^^^^^^^^^^^^^^^^ File "/public/home/user/software/anaconda3/envs/nextpolish/lib/python3.11/site-packages/paralleltask/taskcontrol.py", line 288, in run , stdout, _ = self.run(job.cmd) ^^^^^^^^^^^^^^^^^ File "/public/home/user/software/anaconda3/envs/nextpolish/lib/python3.11/site-packages/paralleltask/task_control.py", line 288, in run log.critical("Command '%s' returned non-zero exit status %d, error info: %s." % (cmd, p.returncode, stderr)) log.critical("Command '%s' returned non-zero exit status %d, error info: %s." % (cmd, p.returncode, stderr)) File "/public/home/user/software/anaconda3/envs/nextpolish/lib/python3.11/logging/init.py", line 1536, in critical File "/public/home/user/software/anaconda3/envs/nextpolish/lib/python3.11/logging/init.py", line 1536, in critical log.critical("Command '%s' returned non-zero exit status %d, error info: %s." % (cmd, p.returncode, stderr)) File "/public/home/user/software/anaconda3/envs/nextpolish/lib/python3.11/logging/init.py", line 1536, in critical self._log(CRITICAL, msg, args, kwargs) self._log(CRITICAL, msg, args, kwargs) File "/public/home/user/software/anaconda3/envs/nextpolish/lib/python3.11/logging/init.py", line 1634, in _log File "/public/home/user/software/anaconda3/envs/nextpolish/lib/python3.11/logging/init.py", line 1634, in _log self._log(CRITICAL, msg, args, **kwargs) File "/public/home/user/software/anaconda3/envs/nextpolish/lib/python3.11/logging/init.py", line 1634, in _log self.handle(record) File "/public/home/user/software/anaconda3/envs/nextpolish/lib/python3.11/logging/init.py", line 1644, in handle self.handle(record) File "/public/home/user/software/anaconda3/envs/nextpolish/lib/python3.11/logging/init.py", line 1644, in handle self.handle(record) File "/public/home/user/software/anaconda3/envs/nextpolish/lib/python3.11/logging/init.py", line 1644, in handle self.callHandlers(record) File "/public/home/user/software/anaconda3/envs/nextpolish/lib/python3.11/logging/init.py", line 1706, in callHandlers self.callHandlers(record) File "/public/home/user/software/anaconda3/envs/nextpolish/lib/python3.11/logging/init.py", line 1706, in callHandlers self.callHandlers(record) File "/public/home/user/software/anaconda3/envs/nextpolish/lib/python3.11/logging/init.py", line 1706, in callHandlers hdlr.handle(record) File "/public/home/user/software/anaconda3/envs/nextpolish/lib/python3.11/logging/init.py", line 978, in handle hdlr.handle(record) File "/public/home/user/software/anaconda3/envs/nextpolish/lib/python3.11/logging/init.py", line 978, in handle hdlr.handle(record) File "/public/home/user/software/anaconda3/envs/nextpolish/lib/python3.11/logging/init.py", line 978, in handle self.emit(record) File "/public/home/user/software/anaconda3/envs/nextpolish/lib/python3.11/site-packages/paralleltask/kit.py", line 42, in emit self.emit(record) File "/public/home/user/software/anaconda3/envs/nextpolish/lib/python3.11/site-packages/paralleltask/kit.py", line 42, in emit self.emit(record) File "/public/home/user/software/anaconda3/envs/nextpolish/lib/python3.11/site-packages/paralleltask/kit.py", line 42, in emit raise Exception(record.msg) raise Exception(record.msg) Exception: Command '/bin/sh /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/04.polish.ref.sh.work/polish_genome2/nextPolish.sh > /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/04.polish.ref.sh.work/polish_genome2/nextPolish.sh.o 2> /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/04.polish.ref.sh.work/polish_genome2/nextPolish.sh.e' returned non-zero exit status 1, error info: . Exception: Command '/bin/sh /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/04.polish.ref.sh.work/polish_genome1/nextPolish.sh > /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/04.polish.ref.sh.work/polish_genome1/nextPolish.sh.o 2> /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/04.polish.ref.sh.work/polish_genome1/nextPolish.sh.e' returned non-zero exit status 1, error info: . raise Exception(record.msg) Exception: Command '/bin/sh /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/04.polish.ref.sh.work/polish_genome3/nextPolish.sh > /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/04.polish.ref.sh.work/polish_genome3/nextPolish.sh.o 2> /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/04.polish.ref.sh.work/polish_genome3/nextPolish.sh.e' returned non-zero exit status 1, error info: . [302167 ERROR] 2024-03-15 15:03:26 polish_genome failed: please check the following logs: [302167 ERROR] 2024-03-15 15:03:26 /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/04.polish.ref.sh.work/polish_genome1/nextPolish.sh.e [302167 ERROR] 2024-03-15 15:03:26 /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/04.polish.ref.sh.work/polish_genome2/nextPolish.sh.e [302167 ERROR] 2024-03-15 15:03:26 /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/04.polish.ref.sh.work/polish_genome3/nextPolish.sh.e

$more /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/04.polish.ref.sh.work/polish_genome1/nextPolish.sh.e hostname

(base) Fri Mar 15 16:48:41 [user@node5 /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish] $more /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/04.polish.ref.sh.work/polish_genome2/nextPolish.sh.e hostname

$more /public/home/user/Plant_genome/assembly/Nextdenove_NextPolish/01_rundir/00.score_chain/04.polish.ref.sh.work/polish_genome2/nextPolish.sh.e hostname

Operating system LSB Version: :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch Distributor ID: CentOS Description: CentOS Linux release 7.6.1810 (Core) Release: 7.6.1810 Codename: Core

GCC $gcc -v Using built-in specs. COLLECT_GCC=gcc COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/4.8.5/lto-wrapper Target: x86_64-redhat-linux Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-linker-hash-style=gnu --enable-languages=c,c++,objc,obj-c++,java,fortran,ada,go,lto --enable-plugin --enable-initfini-array --disable-libgcj --with-isl=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/isl-install --with-cloog=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/cloog-install --enable-gnu-indirect-function --with-tune=generic --with-arch_32=x86-64 --build=x86_64-redhat-linux Thread model: posix gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC)

Python Python 3.11.8

NextPolish nextPolish 1.4.1

To Reproduce (Optional) Steps to reproduce the behavior. Providing a minimal test dataset on which we can reproduce the behavior will generally lead to quicker turnaround time!

Additional context (Optional) Add any other context about the problem here.

moold commented 6 months ago

Could you try this version.