Segmentation Fault

[aakashsur]

I ran into a segmentation fault while running velvetg with k=31. I have 144 million 75 bp paired end Illumina Reads (so 288 million reads total) and 1.2 million long reads. The organism is diploid and fairly repetitive.

I tracked memory usage through the run and that doesn't appear to be the issue. Here is the log velvetg produced:

Wed Jun 14 19:56:45 2017 /home/ubuntu/velvet/velvetg k31 -exp_cov 300 Version 1.2.10 Copyright 2007, 2008 Daniel Zerbino (zerbino@ebi.ac.uk) This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Compilation settings: CATEGORIES = 2 MAXKMERLENGTH = 75 OPENMP LONGSEQUENCES DEBUG

I followed the bug report in the manual, and I've attached my GNU debugger session.

gdb.txt

[dzerbino]

Hello @aakashsur ,

thanks for the bug report.

This is very odd, because the bug suggests a memory failure reading the second element in two fixed size arrays of length 2. Also, this piece of code is used quite intensively, I would not expect a new bug in there.

This smells like memory corruption to me. Possibly due to parallel execution. Would it be feasible for you to try running it without the OpenMP compilation parameter set?

Cheers,

Daniel

[aakashsur]

Daniel,

So I ran it without OpenMP and ran into the seg faults again. I redid everything on Amazon's Linux distro (CentOS based I think?) since I thought it might be an OS thing, or that something was just wrong in my environment.

Here's the compile settings: CATEGORIES = 2 MAXKMERLENGTH = 75 OPENMP LONGSEQUENCES DEBUG

For whatever reason velveth looks like it had different run times and memory usage than when I ran it on Ubuntu. It could just as easily be that I didn't compile something correctly, and I just pipe the "free" command to a log file every second during the run to "track" memory so I don't know how accurate that is.

All the velveth kmer values took around 6 hours to run, and I noticed memory usage hitting the max (total of 480GB of RAM) for some of the kmer values: k=21, used a max of 479GB k=31, used a max of 479GB k=41, used a max of 479GB k=51, used a max of 479GB k=61, used a max of 460GB k=71, used a max of 391GB

Again, not sure if using the "used" value in the "free" command is the best approach to tracking memory usage, but it made me a little uncomfortable that it was using all of the RAM. However none of the runs crashed, and all gave the completed indication in the log. (I've attached a velveth stdout + stderr at the end of the post)

Done inputting sequences Destroying splay table Splay table destroyed

The only reason I mention all this is because I don't know if it's related to the velvetg seg faults. I ran velvetg with the following command ~/velvet/velevtg k21 -exp_cov auto for all the kmer values, and it seg faulted on k=21, k=31.

For k=41 it gave me the following error (I've attached the full log file).

velvetg: Can't calloc 18446744073209251708 KmerOccurences totalling 18446744058200254468 bytes: Cannot allocate memory

Velvetg was able to complete for k=51, k=61, and k=71, but all gave quite low n50 values: k=51 had an n50 of 3,353bp k=61 had an n50 of 3,353bp k=71 had an n50 of 3,810bp though I'm not sure if this has to do with the defaults I used running velvetg, or a memory issue.

To recap, I've attached six files:

1. k=21 velveth log (success)
2. k=21 velveth memory usage (free command every second)
3. k=41 velvetg log (failed)
4. k=41 velvetg memory usage
5. k=71 vevletg log (success)
6. k=71 velveg memory usage

I know there's a lot here, and it doesn't all pertain to the open issue, so let me know if you want to take this offline.

k21.velveth.txt k21.vh-memory.txt k41.velvetg.txt k41.vg-memory.txt k71.velvetg.txt k71.vg-memory.txt

[dzerbino]

Hello @aakashsur , sorry for the slow response, but are you certain you turned off the openmp option? It still shows up in your compilation settings?

[aakashsur]

Hmm, if I get around to it I might try again, thanks for the heads up.