Error: 'std::bad_alloc'

[SwapnilDoijad]

fastaANI ran properly with 100 genomes. However, increased to 1000 genomes resulted in the following error

---

Error details:

---

$ fastANI --ql 1000_genomes.list --rl 1000_genomes.list -o output.txt

Reference = [1.fasta, 2.fasta, ......... 1000.fasta] Query = [1.fasta, 2.fasta, ......... 1000.fasta] Kmer size = 16 Fragment length = 3000 ANI output file = /media/network/project_Lm_all/results/43_fastANI/output.txt

terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc Aborted (core dumped) fastANI --ql 1000_genomes.list --rl 1000_genomes.list -o output.txt

---

Hardware details

---

Processor | 8x Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz Memory | 65858MB Operating System | Ubuntu 16.04.3 LTS

[cjain7]

std::bad_alloc implies FastANI could not allocate memory when it needed. For 1000 microbial genomes, I expect the memory usage to be much below 66G. Please answer few follow up questions here:

1. What is the total size of all genomes you have in in 1000_genomes.list? I wonder if it is too big.

2. Can you provide memory usage of above run? It can be easily obtained by using the /usr/bin/time utility:

/usr/bin/time fastANI --ql 1000_genomes.list --rl 1000_genomes.list -o output.txt

Please make sure you are not running other memory intensive tasks on your system while doing this.

3. Other thing you may want to try is split the reference list (--rl) 1000_genomes.list into two lists of 500 genomes to reduce the memory use and run them one by one as:

fastANI --ql 1000_genomes.list --rl 500_genomes_first.list -o output_1.txt fastANI --ql 1000_genomes.list --rl 500_genomes_second.list -o output_2.txt cat output_1.txt output_2.txt > output.txt

[SwapnilDoijad]

1. 1000_genomes.list contains a list of 1000 genomes, each 3 Mb.

2. After closing all other programs

(A) for 1000 genome

$ /usr/bin/time fastANI --ql 1000_genomes.list --rl 1000_genomes.list -o output.txt Reference = [1.fasta, 2.fasta, ......... 1000.fasta] Query = [1.fasta, 2.fasta, ......... 1000.fasta] Kmer size = 16 Fragment length = 3000 ANI output file = output.txt INFO, skch::Sketch::build, minimizers picked from reference = 305245397 INFO, skch::Sketch::index, unique minimizers = 7172419 INFO, skch::Sketch::computeFreqHist, Frequency histogram of minimizers = (1, 3440253) ... (529726, 1) INFO, skch::Sketch::computeFreqHist, With threshold 0.001%, ignore minimizers occurring >= 2858 times during lookup. INFO, skch::main, Time spent sketching the reference : 286.582 sec INFO, skch::main, Time spent mapping fragments in query #1 : 412.583 sec INFO, skch::main, Time spent post mapping : 20.5822 sec Command terminated by signal 11 648.89user 7.07system 37:54.26elapsed 28%CPU (0avgtext+0avgdata 7770108maxresident)k 0inputs+0outputs (0major+979813minor)pagefaults 0swaps

(B) For 100 genome, each 3 Mb, output is..

successful run

and

7700.57user 1.39system 2:08:32elapsed 99%CPU (0avgtext+0avgdata 1661348maxresident)k 0inputs+3360outputs (0major+438435minor)pagefaults 0swaps

3. creating a bash loop or splitting was the final solution, worked very well.

[cjain7]

Thanks for sharing the info. I tried creating a custom dataset of 1000 E coli genomes at my end but could not reproduce above issue. Let me know if the data you are using is public.