search

vice87 / gam-ngs

Genomic Assemblies Merger for NGS
GNU General Public License v3.0
26 stars 10 forks source link

segmentation fault Linux Ubuntu #2

Open stevebaeyen opened 11 years ago

stevebaeyen commented 11 years ago

Dear Mr., I get a segmentation fault (bug) in the merging phase of GAM-NGS (2 de novo assembly datasets,50bp PE Illumina library. Should I rather use CentOS or Fedora, or do you think there is a bug in Ubuntu ? Best regards, Steve Baeyen

System: Ubuntu Linux 12,04 LTS on Intel® Xeon(R) CPU E5630 @ 2.53GHz × 8 , 16Gb RAM

Make a text file with (sorted/index) .bam file (SAMtools-0.1.19) info BAM files and libraries in : master.PE.bams.txt and slave.PE.bams.txt master.PE.bams.txt vb.: /home/sbaeyen/Bio/GAM-NGS/master-pe-lib1.bam 292 404 slave.PE.bams.txt vb.: /home/sbaeyen/Bio/GAM-NGS/slave-pe-lib1.bam 292 404 BLOCKS CONSTRUCTION sbaeyen@PXLSEQ:~/Bio/GAM-NGS$ gam-create --master-bam master.PE.bams.txt --slave-bam slave.PE.bams.txt --min-block-size 100 --output ngen_clc.prefix [main] opening BAM files [error] master BAM file "$ cat assembly.PE.bams.txt" doesn't exist sbaeyen@PXLSEQ:~/Bio/GAM-NGS$ gam-create --master-bam master.PE.bams.txt --slave-bam slave.PE.bams.txt --minblock- size 100 --output ngen_clc.prefix [main] opening BAM files [error] slave BAM file "/home/sbaeyen/Bio/GAM-NGS/master-pe-lib2.bam" doesn't exist sbaeyen@PXLSEQ:~/Bio/GAM-NGS$ gam-create --master-bam master.PE.bams.txt --slave-bam slave.PE.bams.txt --minblock- size 100 --output ngen_clc.prefix [main] opening BAM files [bam] ERROR: unable to open BAM index file: /home/sbaeyen/Bio/GAM-NGS/master-pe-lib1.bam.bai sbaeyen@PXLSEQ:~/Bio/GAM-NGS$ gam-create --master-bam master.PE.bams.txt --slave-bam slave.PE.bams.txt --minblock- size 100 --output ngen_clc.prefix [main] opening BAM files [main] loading reads in memory [main] reads loaded in 43s [main] finding blocks [main] blocks found = 1654 [main] writing blocks on file: ngen_clc.prefix [main] total execution time = 1m11s Segmentatiefout (geheugendump gemaakt) MERGE BLOCKS gam-merge --master-bam --slave-bam --blocks-file

--master-fasta --slave-fasta --min-block-size --threads --output 2> merge.err sbaeyen@PXLSEQ:~/Bio/GAM-NGS$ gam-merge --master-bam master.PE.bams.txt --slave-bam slave.PE.bams.txt --blocks-file ngen_clc.prefix.blocks --master-fasta LMG19145_Ngen_v11_.fasta --slave-fasta LMG19145_Geneious.fasta --min-block-size 100 --threads 8 --output LMG19145_merge_ngen_CLC 2> merge.err [main] Loading blocks [main] Loaded blocks = 1654 [main] Loading BAMs data [bam] Master PE-alignments file master.PE.bams.txt successfully opened: /home/sbaeyen/Bio/GAM-NGS/master-pe-lib1.bam inserts size = 342.689 +/- 19.0181 coverage = 76.8512 [bam] Slave PE-alignments file slave.PE.bams.txt successfully opened: /home/sbaeyen/Bio/GAM-NGS/slave-pe-lib2.bam inserts size = 342.648 +/- 19.0005 coverage = 77.0297 [main] Loading contigs data [main] Filtering blocks by coverage [main] Remaining blocks = 337 [main] Partitioning blocks [main] Loading contig sequences [merge] Writing contigs with no blocks to file: LMG19145_merge_ngen_CLC.noblocks.BF.fasta Segmentatiefout (geheugendump gemaakt) # segmentation fault (memory dump) Info Ubuntu: Segmentation fault is caused by a bug in the application. Technically it means that application try to read or write to part of memory that doesn't belongs to it (or doesn't exist). It's of course forbidden to read or write to somebody else's memory and when system (kernel) detects this, it will force the application to quit.
guyleonard commented 11 years ago

Seg faults aren't specific to Ubuntu over any other flavour of linux. Often I find they occur when you run out of memory running whatever it is you are doing - now that might be due to not enough memory on your system or perhaps due to a problem in the GAM-NGS code...

guyleonard commented 11 years ago

Hmm I too get this error!

[main] opening BAM files [main] loading reads in memory [main] reads loaded in 3h6m43s [main] finding blocks [main] blocks found = 259670 [main] writing blocks on file: ray_newbler_50 [main] total execution time = 4h43m5s Segmentation fault (core dumped)

Linux 3.2.0-23-generic #36-Ubuntu SMP. 2x 8 core Intel, 512GB RAM.

Two assemblies, with two sets of PE-reads one from HiSeq and one from GAIIx

As far as I can tell I created the .bam files correctly by aligning with 'bwa mem', then sorting them and then indexing them. The config files read as below:

/ray_K31/ray_assembly_hiseq_pe_proper_sorted.bam 184 272 /ray_K31/ray_assembly_gaiix_s1_pe_proper_sorted.bam 113 371 /ray_K31/ray_assembly_gaiix_s2_pe_proper_sorted.bam 113 371

The HiSeq is 40GB and the GAIIx are around 2G. So, there should be plenty of RAM. Indeed, this even occurs when only using the smaller GAIIx runs... :(

vice87 commented 11 years ago

@stevebaeyen I've updated the repository with the latest version of GAM-NGS. I'm not sure your problem has been fixed. However, if it isn't solved yet, you should get more output just before the segmentation fault. That output should let me understand better where there might be the bug.

@guyleonard If you mean the "Segmentation fault" before gam-create command terminates, it has been corrected with the latest version I've just committed to the repository. However, that error doesn't affect GAM-NGS's output which should have been created correctly (you can proceed with gam-merge, using the file ray_newbler_50.blocks).

guyleonard commented 11 years ago

Meant to reply earlier, awesome that latest version you pushed removed all the Seg Faults! Cheers! :)

olekto commented 11 years ago

Hi Riccardo, I'm also getting a seg fault with the newer git source (pulled a couple of days ago). This is the output from gam-merge: [main] Loading blocks [main] Loaded blocks = 3214732 [main] Loading BAMs data [bam] Master PE-alignments file 130827_newb.bams.txt successfully opened: /work/olekto/cod2_asms/gam-ngs/130827_454_BACends_refill_het/130827_newbler_300PE.sorted.bam inserts size = 309.709 +/- 18.8646 coverage = 100.957 /work/olekto/cod2_asms/gam-ngs/130827_454_BACends_refill_het/130827_newbler_5k.sorted.bam inserts size = 4967.78 +/- 747.126 coverage = 0 [bam] Slave PE-alignments file 454_ilm_mT.bams.txt successfully opened: /work/olekto/cod2_asms/gam-ngs/454_ilm_mT/454_ilm_mT_300PE.sorted.bam inserts size = 309.937 +/- 18.4968 coverage = 112.567 /work/olekto/cod2_asms/gam-ngs/454_ilm_mT/454_ilm_mT_5k.sorted.bam inserts size = 4833.27 +/- 708.529 coverage = 0 [main] Loading contigs data [main] Filtering blocks by coverage [main] Remaining blocks = 183180 [main] Partitioning blocks

I turned on graphs, and it seems to output all the contig stuff (from partitionBlocksByPairedContigs), but stops after this.

Do you have any idea what might be wrong?

kstatebioinfo commented 11 years ago

I am also getting a segmentation fault. How much memory do you suspect I might need for a 200 Mb genome? I have tried 30G, 40G, and I have just queued 100G. So far I get a segmentation fault and core dump each time even though my max reported memory was ~5G. Does anyone have a suggestion of what could cause this? Below is my standard error file and resource usage notes:

standard error: /opt/sge/default/spool/mage09/job_scripts/1305051: line 4: 6717 Segmentation fault (core dumped) /homes/bioinfo/bioinfo_software/gam-ngs/bin/gam-create --master-bam /homes/bioinfo/Tcas/jobs/master.tcas4.0.PE.bams.txt --slave-bam /homes/bioinfo/Tcas/jobs/slave.sacffolds.PE.bams.txt --min-block-size 10 --output master.tcas4.0.slave.sacffolds

resource usage: Job 1305051 (run_gam-ngs_scaffolds.sh) Complete User = bioinfo Queue = highmem.q@mage09.beocat Host = mage09.beocat Start Time = 10/04/2013 14:49:21 End Time = 10/04/2013 15:09:41 User Time = 00:07:20 System Time = 00:00:50 Wallclock Time = 00:20:20 CPU = 00:08:11 Max vmem = 4.677G Exit Status = 139

They fault occurred after [main] opening BAM files and [main] loading reads in memory.

Thanks, Jennifer

kstatebioinfo commented 11 years ago

Would it be possible to add a example dataset to the github repo so that I could test my installation?

vice87 commented 11 years ago

I'll add a example data set as soon as possible. About the segmentation fault: what is the content of the files master.tcas4.0.PE.bams.txt and slave.sacffolds.PE.bams.txt?

kstatebioinfo commented 11 years ago

master.tcas4.0.PE.bams.txt is below /homes/bioinfo/Tcas/bowtie2_alignments/tca_ref_chrLG_all_8kb_unambig_sorted.bam 5000 9000 /homes/bioinfo/Tcas/bowtie2_alignments/tca_ref_chrLG_all_20kb_unambig_sorted.bam 17000 30000 /homes/bioinfo/Tcas/bowtie2_alignments/tca_ref_chrLG_all_3kb_unambig_sorted.bam 1000 5000 slave.sacffolds.PE.bams.txt is below: /homes/bioinfo/Tcas/bowtie2_alignments/Kmer_merges_81-scaffolds_20kb_unambig_sorted.bam 17000 30000 /homes/bioinfo/Tcas/bowtie2_alignments/Kmer_merges_81-scaffolds_3kb_unambig_sorted.bam 1000 5000 /homes/bioinfo/Tcas/bowtie2_alignments/Kmer_merges_81-scaffolds_8kb_unambig_sorted.bam 5000 9000

kstatebioinfo commented 11 years ago

The ranges are based on histograms generated by ABySS.

kstatebioinfo commented 11 years ago

Thanks for offering a test dataset. I think it would help to be able to check my input against input that is know to work.

-Jennifer

natassagioti commented 9 years ago

I get the same error, too, is this issue solved? I am running gam-merge on a 4-core MacOS 10.9.5 with 16G of memory, and i get a seg fault in both directions (slave/master), although the output is slightly different:

1. /Applications/myinstalledprogs/gam-ngs-master/bin/gam-merge --master-bam combinedMummer.PE.bams.txt --slave-bam cleanCLC.PE.bams.txt --blocks-file SPADes2CLC-combo.blocks --master-fasta combined-cleanFull_MUmmer.fasta --slave-fasta clean_CLC.fasta --min-block-size 10 --output SPADesmasterCLC-combo 2> SPADesmasterCLC-combo.err [main] Loading blocks [main] Loaded blocks = 8671 [main] Loading BAMs data [bam] Master PE-alignments file combinedMummer.PE.bams.txt successfully opened: /Volumes/Data/Dakos/assembly/tocombine/genome-contigs/SPAdescombined_CLC/GAM/combinedcleanFull-MUMMER_5.bam inserts size = 400.833 +/- 92.7029 coverage = 93.8782 /Volumes/Data/Dakos/assembly/tocombine/genome-contigs/SPAdescombined_CLC/GAM/combinedcleanFull-MUMMER_7.bam inserts size = 373.498 +/- 89.1039 coverage = 688.913 /Volumes/Data/Dakos/assembly/tocombine/genome-contigs/SPAdescombined_CLC/GAM/combinedcleanFull-MUMMER_8.bam inserts size = 375.237 +/- 91.5148 coverage = 687.61 [bam] Slave PE-alignments file cleanCLC.PE.bams.txt successfully opened: /Volumes/Data/Dakos/assembly/tocombine/genome-contigs/SPAdescombined_CLC/GAM/clean_CLC.fasta_5.bam inserts size = 388.246 +/- 100.588 coverage = 163.67 /Volumes/Data/Dakos/assembly/tocombine/genome-contigs/SPAdescombined_CLC/GAM/clean_CLC.fasta_7.bam inserts size = 350.418 +/- 107.09 coverage = 1189.87 /Volumes/Data/Dakos/assembly/tocombine/genome-contigs/SPAdescombined_CLC/GAM/clean_CLC.fasta_8.bam inserts size = 351.314 +/- 109.466 coverage = 1184.92 [main] Loading contigs data...done. Master Assembly: sequences = 1096 total length = 4021893 Slave Assembly: sequences = 644 total length = 2083754 [main] Retrieving contigs without blocks...done. [main] Filtering blocks by coverage [main] Remaining blocks = 620 [main] Retrieving contigs with no blocks after coverage filtering...done. [main] Partitioning blocks [main] Loading contig sequences master sequences loaded = 1096 slave sequences loaded = 644 [merge] Writing contigs with no blocks to file: SPADesmasterCLC-combo.noblocks.BF.fasta Segmentation fault: 11

2. postdoc1:GAM natassa$ /Applications/myinstalledprogs/gam-ngs-master/bin/gam-merge --master-bam cleanCLC.PE.bams.txt --slave-bam combinedMummer.PE.bams.txt --blocks-file SPADes2CLCmaster-combo.blocks --master-fasta clean_CLC.fasta --slave-fasta combined-cleanFull_MUmmer.fasta --min-block-size 10 --output SPADesCLCmaster-combo 2> SPADesCLCmaster-combo.err [main] Loading blocks [main] Loaded blocks = 8671 [main] Loading BAMs data [bam] Master PE-alignments file cleanCLC.PE.bams.txt successfully opened: /Volumes/Data/Dakos/assembly/tocombine/genome-contigs/SPAdescombined_CLC/GAM/clean_CLC.fasta_5.bam inserts size = 388.246 +/- 100.588 coverage = 163.67 /Volumes/Data/Dakos/assembly/tocombine/genome-contigs/SPAdescombined_CLC/GAM/clean_CLC.fasta_7.bam inserts size = 350.418 +/- 107.09 coverage = 1189.87 /Volumes/Data/Dakos/assembly/tocombine/genome-contigs/SPAdescombined_CLC/GAM/clean_CLC.fasta_8.bam inserts size = 351.314 +/- 109.466 coverage = 1184.92 [bam] Slave PE-alignments file combinedMummer.PE.bams.txt successfully opened: /Volumes/Data/Dakos/assembly/tocombine/genome-contigs/SPAdescombined_CLC/GAM/combinedcleanFull-MUMMER_5.bam inserts size = 400.833 +/- 92.7029 coverage = 93.8782 /Volumes/Data/Dakos/assembly/tocombine/genome-contigs/SPAdescombined_CLC/GAM/combinedcleanFull-MUMMER_7.bam inserts size = 373.498 +/- 89.1039 coverage = 688.913 /Volumes/Data/Dakos/assembly/tocombine/genome-contigs/SPAdescombined_CLC/GAM/combinedcleanFull-MUMMER_8.bam inserts size = 375.237 +/- 91.5148 coverage = 687.61 [main] Loading contigs data...done. Master Assembly: sequences = 644 total length = 2083754 Slave Assembly: sequences = 1096 total length = 4021893 [main] Retrieving contigs without blocks...done. [main] Filtering blocks by coverage [main] Remaining blocks = 619 [main] Retrieving contigs with no blocks after coverage filtering...done. [main] Partitioning blocks [main] Loading contig sequences Segmentation fault: 11

I previously run gam-merge without error, the fasta files used were 2.6 and 3.5 M, the files used now are not significantly bigger (2.2 and 4.1 M). Note that in the previous successful run, I did not get any indication of the memory used ([main] MAX Memory used: 0.00 KB), and I considered this was a minor bug. Thank you, Natassa