Excesive memory consumption whilst precalculating kmers

iqbal-lab-org / gramtools

Genome inference from a population reference graph

MIT License

92 stars 15 forks source link

Excesive memory consumption whilst precalculating kmers #50

Closed ffranr closed 6 years ago

ffranr commented 7 years ago

Precalculating kmers takes an excessive amount of memory.

RAM memory usage peaked at ~32MiB whilst calculating the ksize_9.precalc kmers file for the test1 dataset. See attached file test1_massif.out.12449.zip

This is unlikely due to the thread creation routine. Exiting just after thread creation showed low memory usage on the WG dataset. Memory usage seems to increase quickly as the threads are running.

ffranr commented 6 years ago

I can now generate the kmer index on my laptop without a noticeable performance impact.