search

simoncchu / GAPPadder

GAPPadder is tool for closing gaps on draft genomes with short sequencing data
27 stars 7 forks source link

assembly not running #1

Closed pganot closed 7 years ago

pganot commented 7 years ago

Hi

I have tried to run gappdder on my draft genome with a sample (2X 1000000 reads) of a miseq library, as a program-test. The 2 first steps (Preprocess & Collect) went apparently OK, However, it crushed during the assembly step. In my log file, I have a lot of similar error messages(14Mb text file) recurrently : [E::fai_insert_index] Malformed line [E::bwa_idx_load] fail to locate the index files [bns_fasta2bntseq] Failed to allocate 0 bytes at bntseq.c line 270: Success

Below is the part of the log file.

Running command: /media/physio/home/pganot/finisher/GAPPadder/ContigsMerger -s 0.4 -i1 -2.0 -i2 -2.0 -x 12 -y 50 -k 10 -t 5 -m 1 -o /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/579_1/contigs.fa_no_dup.fa.merge.info /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/579_1/contigs.fa_no_dup.fa > /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/579_1/contigs.fa_no_dup.fa.merged.fa ... Running command: /media/physio/home/pganot/finisher/GAPPadder/TERefiner_1 -P -b /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/579_1/contigs.fa_no_dup.fa.merged.fa.itself.sort.bam -r /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/579_1/contigs.fa_no_dup.fa.merged.fa -o /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/579_1/contigs.fa_no_dup.fa.merged.fa.no_dup.fa -c 0.99 -g ... Running command: /media/physio/home/pganot/finisher/GAPPadder/TERefiner_1 -P -b /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/579_1/contigs.fa_no_dup.fa.merged.fa.no_dup.fa.itself.sort.bam -r /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/579_1/contigs.fa_no_dup.fa.merged.fa.no_dup.fa -o /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/579_1/contigs.fa_no_dup.fa.merged.fa.no_dup.fa.no_contained.fa -c 0.99 ... Running command: /media/physio/home/pganot/finisher/GAPPadder/TERefiner_1 -P -b /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/597_2/contigs.fa.itself.sort.bam -r /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/597_2/contigs.fa -o /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/597_2/contigs.fa_no_dup.fa -c 0.99 -g ... Running command: /media/physio/home/pganot/finisher/GAPPadder/ContigsMerger -s 0.4 -i1 -2.0 -i2 -2.0 -x 12 -y 50 -k 10 -t 5 -m 1 -o /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/597_2/contigs.fa_no_dup.fa.merge.info /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/597_2/contigs.fa_no_dup.fa > /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/597_2/contigs.fa_no_dup.fa.merged.fa ... Running command: /media/physio/home/pganot/finisher/GAPPadder/TERefiner_1 -P -b /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/597_2/contigs.fa_no_dup.fa.merged.fa.itself.sort.bam -r /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/597_2/contigs.fa_no_dup.fa.merged.fa -o /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/597_2/contigs.fa_no_dup.fa.merged.fa.no_dup.fa -c 0.99 -g ... Running command: /media/physio/home/pganot/finisher/GAPPadder/TERefiner_1 -P -b /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/597_2/contigs.fa_no_dup.fa.merged.fa.no_dup.fa.itself.sort.bam -r /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/597_2/contigs.fa_no_dup.fa.merged.fa.no_dup.fa -o /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/597_2/contigs.fa_no_dup.fa.merged.fa.no_dup.fa.no_contained.fa -c 0.99 ... Running command: /media/physio/home/pganot/finisher/GAPPadder/TERefiner_1 -P -b /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/618_2/contigs.fa.itself.sort.bam -r /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/618_2/contigs.fa -o /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/618_2/contigs.fa_no_dup.fa -c 0.99 -g ... Running command: /media/physio/home/pganot/finisher/GAPPadder/ContigsMerger -s 0.4 -i1 -2.0 -i2 -2.0 -x 12 -y 50 -k 10 -t 5 -m 1 -o /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/618_2/contigs.fa_no_dup.fa.merge.info /media/physio/home/pganot/finisher/GAPPadder/v143_ssparm: cannot remove /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/613_2/contigs.fa_no_dup.fa.merged.fa.fai': No such file or directory [E::fai_insert_index] Malformed line Could not build fai index /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/616_1/contigs.fa_no_dup.fa.merged.fa.fai [E::bwa_idx_load] fail to locate the index files [bwa_index] Pack FASTA... [bns_fasta2bntseq] Failed to allocate 0 bytes at bntseq.c line 270: Success [E::fai_insert_index] Malformed line Could not build fai index /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/613_2/contigs.fa_no_dup.fa.merged.fa.no_dup.fa.fai Could not build fai index /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/626_4/contigs.fa.fai [E::fai_insert_index] Malformed line Could not build fai index /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/615_4/contigs.fa_no_dup.fa.merged.fa.fai rm: cannot remove/media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/607_2/contigs.fa_no_dup.fa.merged.fa.no_dup.fa.sa': No such file or directory rm: cannot remove /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/604_4/contigs.fa_no_dup.fa.merged.fa.no_dup.fa.fai': No such file or directory [E::fai_insert_index] Malformed line Could not build fai index /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/617_7/contigs.fa_no_dup.fa.merged.fa.fai [E::bwa_idx_load] fail to locate the index files [E::fai_insert_index] Malformed line Could not build fai index /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/618_2/contigs.fa_no_dup.fa.merged.fa.fai [bwa_index] Pack FASTA... [bns_fasta2bntseq] Failed to allocate 0 bytes at bntseq.c line 270: Success Could not build fai index /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/626_4/contigs.fa.fai [E::fai_insert_index] Malformed line Could not build fai index /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/613_2/contigs.fa_no_dup.fa.merged.fa.no_dup.fa.fai rm: cannot remove/media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/607_2/contigs.fa_no_dup.fa.merged.fa.no_dup.fa.bwt': No such file or directory [E::fai_insert_index] Malformed line Could not build fai index /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/615_4/contigs.fa_no_dup.fa.merged.fa.fai [E::bwa_idx_load] fail to locate the index files [bwa_index] Pack FASTA... rm: cannot remove /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/607_1/contigs.fa_no_dup.fa.merged.fa.no_dup.fa.fai'[bwa_index] Pack FASTA... : No such file or directory rm: cannot remove/media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/610_3/contigs.fa_no_dup.fa.merged.fa.fai': No such file or directory [bns_fasta2bntseq] Failed to allocate 0 bytes at bntseq.c line 270: Success [bns_fasta2bntseq] Failed to allocate 0 bytes at bntseq.c line 270: Success rm: cannot remove /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/607_2/contigs.fa_no_dup.fa.merged.fa.no_dup.fa.ann': No such file or directory [bwa_index] Pack FASTA... [E::fai_insert_index] Malformed line Could not build fai index /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/618_2/contigs.fa_no_dup.fa.merged.fa.fai 0.00 sec [bwa_index] Construct BWT for the packed sequence... rm: cannot remove/media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/607_2/contigs.fa_no_dup.fa.merged.fa.no_dup.fa.amb': No such file or directory [E::fai_build3] Failed to open the FASTA file /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/610_3/contigs.fa_no_dup.fa.merged.fa.no_dup.fa Could not build fai index /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/610_3/contigs.fa_no_dup.fa.merged.fa.no_dup.fa.fai [bwa_index] 0.00 seconds elapse. [bwa_index] Update BWT... [E::bwa_idx_load] fail to locate the index files 0.00 sec [bwa_index] Pack forward-only FASTA... [E::bwa_idx_load] fail to locate the index files [bwa_index] Pack FASTA... rm: cannot remove /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/620_2/contigs.fa.sa': No such file or directory rm: cannot remove/media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/622_3/contigs.fa.sa': No such file or directory rm: cannot remove /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/614_1/contigs.fa_no_dup.fa.merged.fa.sa': No such file or directory [E::bwa_idx_load] fail to locate the index files 0.00 sec [bwa_index] Construct SA from BWT and Occ... 0.00 sec [main] Version: 0.7.10-r789 [main] CMD: /media/physio/build/bwa-0.7.10/bwa index /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/615_4/contigs.fa_no_dup.fa.merged.fa [main] Real time: 0.018 sec; CPU: 0.009 sec [E::fai_build3] Failed to open the FASTA file /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/610_3/contigs.fa_no_dup.fa.merged.fa.no_dup.fa Could not build fai index /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/610_3/contigs.fa_no_dup.fa.merged.fa.no_dup.fa.fai rm: cannot remove/media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/620_2/contigs.fa.bwt': No such file or directory rm: cannot remove /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/610_1/contigs.fa_no_dup.fa.merged.fa.no_dup.fa.sa': No such file or directory [bwa_index] Pack FASTA... rm: cannot remove/media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/614_1/contigs.fa_no_dup.fa.merged.fa.bwt': No such file or directory rm: cannot remove /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/622_3/contigs.fa.bwt': No such file or directory [bwa_index] fail to open file '/media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/610_3/contigs.fa_no_dup.fa.merged.fa.no_dup.fa' : No such file or directory [M::main_mem] read 1 sequences (188 bp)... rm: cannot remove/media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/620_2/contigs.fa.ann': No such file or directory [bwa_index] Pack FASTA... [M::mem_process_seqs] Processed 1 reads in 0.002 CPU sec, 0.001 real sec [main] Version: 0.7.10-r789 [main] CMD: /media/physio/build/bwa-0.7.10/bwa mem -a /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/615_4/contigs.fa_no_dup.fa.merged.fa /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/615_4/contigs.fa_no_dup.fa.merged.fa [main] Real time: 0.002 sec; CPU: 0.003 sec 0.00 sec [bwa_index] Construct BWT for the packed sequence... rm: cannot remove /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/614_1/contigs.fa_no_dup.fa.merged.fa.ann': No such file or directory rm: cannot remove/media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/622_3/contigs.fa.ann': No such file or directory rm: cannot remove `/media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/615_2/contigs.fa_no_dup.fa.merged.fa.sa': No such file or directory

the end of log file: [main] CMD: /media/physio/build/bwa-0.7.10/bwa index /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/5052_1/contigs.fa [main] Real time: 0.005 sec; CPU: 0.007 sec [E::bwa_idx_load] fail to locate the index files [M::main_mem] read 2 sequences (590 bp)... [E::bwa_idx_load] fail to locate the index files [bwa_index] Construct BWT for the packed sequence... [bwa_index] Pack FASTA... [bns_fasta2bntseq] Failed to allocate 0 bytes at bntseq.c line 270: Success [bwa_index] 0.00 seconds elapse. [bwa_index] Update BWT... [bwa_index] Pack FASTA... 0.00 sec [bwa_index] Pack forward-only FASTA... [bns_fasta2bntseq] Failed to allocate 0 bytes at bntseq.c line 270: Success [M::mem_process_seqs] Processed 2 reads in 0.004 CPU sec, 0.003 real sec [main] Version: 0.7.10-r789 [main] CMD: /media/physio/build/bwa-0.7.10/bwa mem -T 30 -a /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/4682_1/contigs.fa /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/../flank_regions/4682_1.fa [main] Real time: 0.003 sec; CPU: 0.004 sec 0.00 sec [bwa_index] Construct SA from BWT and Occ... [E::bwa_idx_load] fail to locate the index files [E::bwa_idx_load] fail to locate the index files [bwa_index] Pack FASTA... 0.00 sec [main] Version: 0.7.10-r789 [main] CMD: /media/physio/build/bwa-0.7.10/bwa index /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/5109_1/contigs.fa [main] Real time: 0.005 sec; CPU: 0.008 sec [bns_fasta2bntseq] Failed to allocate 0 bytes at bntseq.c line 270: Success [bwa_index] Pack FASTA... [bns_fasta2bntseq] Failed to allocate 0 bytes at bntseq.c line 270: Success [M::main_mem] read 2 sequences (590 bp)... [M::main_mem] read 2 sequences (590 bp)... [bwa_index] Pack FASTA... [bns_fasta2bntseq] Failed to allocate 0 bytes at bntseq.c line 270: Success [bwa_index] Pack FASTA... [M::mem_process_seqs] Processed 2 reads in 0.002 CPU sec, 0.002 real sec [bns_fasta2bntseq] Failed to allocate 0 bytes at bntseq.c line 270: Success [main] Version: 0.7.10-r789 [main] CMD: /media/physio/build/bwa-0.7.10/bwa mem -T 30 -a /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/4956_1/contigs.fa /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/../flank_regions/4956_1.fa [main] Real time: 0.002 sec; CPU: 0.003 sec [bwa_index] Pack FASTA... [E::bwa_idx_load] fail to locate the index files [bns_fasta2bntseq] Failed to allocate 0 bytes at bntseq.c line 270: Success [E::bwa_idx_load] fail to locate the index files [M::mem_process_seqs] Processed 2 reads in 0.002 CPU sec, 0.002 real sec [bwa_index] Pack FASTA... [main] Version: 0.7.10-r789 [main] CMD: /media/physio/build/bwa-0.7.10/bwa mem -T 30 -a /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/5052_1/contigs.fa /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/../flank_regions/5052_1.fa [main] Real time: 0.002 sec; CPU: 0.003 sec [bns_fasta2bntseq] Failed to allocate 0 bytes at bntseq.c line 270: Success [bwa_index] Pack FASTA... [bns_fasta2bntseq] Failed to allocate 0 bytes at bntseq.c line 270: Success [E::bwa_idx_load] fail to locate the index files [M::main_mem] read 2 sequences (590 bp)... [E::bwa_idx_load] fail to locate the index files [M::mem_process_seqs] Processed 2 reads in 0.003 CPU sec, 0.002 real sec [main] Version: 0.7.10-r789 [main] CMD: /media/physio/build/bwa-0.7.10/bwa mem -T 30 -a /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/velvet_temp/5109_1/contigs.fa /media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/../flank_regions/5109_1.fa [main] Real time: 0.002 sec; CPU: 0.003 sec [E::bwa_idx_load] fail to locate the index files [E::bwa_idx_load] fail to locate the index files [E::bwa_idx_load] fail to locate the index files [E::bwa_idx_load] fail to locate the index files [E::bwa_idx_load] fail to locate the index files Traceback (most recent call last): File "/media/physio/home/pganot/finisher/GAPPadder/main.py", line 283, in main_func(scommand,sfconfig) File "/media/physio/home/pganot/finisher/GAPPadder/main.py", line 274, in main_func gap_assembler.assemble_pipeline() File "/media/physio/home/pganot/finisher/GAPPadder/assemble_gaps.py", line 339, in assemble_pipeline id_remain=self.pick_already_constructed(contigs_select, fa_list, sf_picked) File "/media/physio/home/pganot/finisher/GAPPadder/assemble_gaps.py", line 321, in pick_already_constructed m_picked=contigs_select.get_already_picked(sf_picked) File "/media/physio/home/pganot/finisher/GAPPadder/pick_contigs.py", line 576, in get_already_picked with open(sf_picked) as fin_picked: IOError: [Errno 2] No such file or directory: u'/media/physio/home/pganot/finisher/GAPPadder/v143_sspace/merged/../picked_seqs.fa'

I have tried to recompile TERefiner, but I got the folling error file after make: g++ -O3 -Wall -static -I/media/physio/build/bamtools/include -L/media/physio/build/bamtools/lib -Wl,-rpath,/media/physio/build/bamtools/lib -o TERefiner_1 public_func.o StrOperation.o local_alignment.o Alignment.o bam_parse.o fai_parser.o contigs.o Coverage.o scaffolding.o RepeatsClassifier.o refiner.o main.o -lbamtools -lz -lm /usr/bin/ld: cannot find -lz collect2: ld returned 1 exit status make: *** [TERefiner] Error 1

Could you help ? Thanks

pganot commented 7 years ago

I have now managed to compile TERefiner. It needed zlib-static and glibc-static libraries (not included in basic or -devel packages from Centos) . I am currently re-running GAPPadder and thus far, it is running through the assembly step. Closing issue and crossing fingers!