search

dzerbino / velvet

Short read de novo assembler using de Bruijn graphs, as published in: D.R. Zerbino and E. Birney. 2008. Velvet: algorithms for de novo short read assembly using de Bruijn graphs. Genome Research, 18: 821-829
https://europepmc.org/article/pmc/2336801
GNU General Public License v2.0
278 stars 99 forks source link

Velveth Killed #34

Closed aakashsur closed 7 years ago

aakashsur commented 7 years ago

I tried doing

velveth data/ 71 -shortPaired -fasta -separate R1.fasta R2.fasta -long PacBio.fasta which has about 144 million 75 bp paired end Illumina Reads (so 288 million reads total) and 1.2 million long reads.

However at the end of the process, it just says "Killed". Is this because the memory ran out? Specifically, the last line is:

[3521.119679] Inputting sequence 288000000 / 289697228 Killed

I ran the process on a machine with 60GB or RAM, and I'm retrying it on a machine with 256GB or RAM. Is this a memory issue, if so, how much do I need?

dzerbino commented 7 years ago

Hello @aakashsur,

Killed means that is was actively killed by another system, typically a job scheduler (LSF, SGE...) or the OOM killer of the system. This is generally associated to memory, but not always.

It's odd that it got killed so close to the end though.

HTH,

Daniel

aakashsur commented 7 years ago

Hi,

Thanks for following up. I tried lowering the k-mer value to see if that would reduce the memory requirement, and I ran it with 200GB instead of 60GB, and this time on a SLURM cluster.

With k=31 """ [18365.393775] Inputting sequence 285000000 / 289697228 [18424.194206] Inputting sequence 286000000 / 289697228 [18487.424620] Inputting sequence 287000000 / 289697228 [18550.339204] Inputting sequence 288000000 / 289697228 slurmstepd: Job 26438 exceeded memory limit (204803480 > 204800000), being killed slurmstepd: Exceeded job memory limit slurmstepd: JOB 26438 CANCELLED AT 2017-06-03T01:33:23 slurmstepd: Exceeded step memory limit at some point. Step may have been partially swapped out to disk. """ I also tried k=41, 51, 61, and 71, but they all (predictably) failed with a similar message. Is there some big memory intensive step at the end of the process?

What kind of memory do you think I need?

dzerbino commented 7 years ago

It's very hard to tell without knowing all the parameters of the sequencing run. There used to be an app that estimated this but it seems to have been discontinued.

aakashsur commented 7 years ago

Well I put it on a bigger machine, and it looks like I was able to have it finish. I tracked time to completion and memory usage through the process. With 144 million 75 bp paired end Illumina Reads (so 288 million reads total) and 1.2 million long reads, the various k-mer values, max memory usage, and completion times were:

k-21: 292GB, 15 hours 15 minutes k-31: 322GB, 10 hours 30 minutes k-41: 326GB, 15 hours 41 minutes k-51: 322GB, 11 hours 12 minutes k-61: 313GB, 12 hours 11 minutes k-71: 301GB, 14 hours 13 minutes

One thing I noticed was that the vast majority of the time and memory usage happened after the last of the stdout messages about "inputting sequences". The roadmaps file kept growing during the remaining hours. Also, I was curious, does multithreading the install make velveth any faster?

dzerbino commented 7 years ago

Glad it worked in the end!