Closed dombraccia closed 4 years ago
Thank you @dombraccia , I think unfortunately this is an issue that happens when we have a high number of distinct kmers regardless of the size of the fasta file. I think in such cases even choosing the large number for the bloomfilter based on the approximate kmer distribution, still the final hash-table is gonna be so large that we still need to deal with the slowness and growth in memory caused by it.
I think in those cases it might be worth trying to find the sweet spot between the size of the final hash-table and the bloom filter.
What do you think @rob-p ?
Update: Pufferfish ended up finishing the index! It took ~3.3 days in total, but it finished.
I will close the issue, but I'll also run the indexing step again with a bigger bloom filter size of 2^38 or 2^39 bits and report back the time here to potentially create a better recommendation of filter size of reference material.
Hello COMBINE-lab, I am trying to use salmon to quantify genes from a gene catalogue (Integrative Gene Catalogue for human gut microbe genes) using shotgun reads from HMP2, but my pipeline seems to be stuck at the index building phase, specifically, the twopaco step of the index. The only line of code I have run so far is:
salmon index -t data/IGC/IGC.fa -i processed/IGC_index --threads 16 -f 37
where
IGC.fa
is a fasta file with ~9,870,000 genes, and takes up 7.7gb (ntHll estimated 6,091,444,578 distinct k-mers). the recommended filter size was 2^37, so on a subsequent run, I just included the-f 37
flag to save time calculating that.The only issue now is that it seems to be stuck at this building step for a while (program has been running for 24 hrs) so I wanted to check if that was normal, or if I should modify my parameters to somehow get past this point. I am running this on a HPC and I requested 500gb of RAM, so I don't think memory is the problem.
Here is the current state of the error log:
And here is the current state of the output log:
Let me know if there is anything else I should provide you with