search

simonrharris / SKA

Split Kmer Analysis
MIT License
62 stars 1 forks source link

Less memory for `merge` and `align` #4

Open bewt85 opened 6 years ago

bewt85 commented 6 years ago

We just has a chat about this. I was trying to merge two kmerge files of about 450-500MB each. I ran out of memory on a 32GB machine. I have a feeling you could do merging with a lot less memory if you did some memory mapping or if you held the kmers in memory but only used a reference to the bitstring for the alleles.

You might be able to do something similar to make the align use a lot less memory but I'm less sure.

foamy1881 commented 4 years ago

I'm having the same issue here and ran out of memory during the merging process. Is a 32 GB VM .

tseemann commented 4 years ago

Code: https://github.com/simonrharris/SKA/blob/master/src/sk_merge.cpp