Open diego-rt opened 10 months ago
@diego-rt Sorry for the late reply:
40
?But we are aware that certain sections of the code can be optimized to reduce hifiasm's memory requirements. Do you need to run it urgently? I can cut a release to reduce the memory requirement.
Hi @chhylp123
No worries, thanks a lot for your reply as always!
Actually that would be incredible! At the moment its taking roughly 10 days to run (even with 88 cores) only to run out of memory during the last correction round, so we can't really run it on our full dataset yet.
Thanks a lot for your help once again!
Thanks. I will cut a new release soon.
But right now you may also consider to use less number of CPUs and smaller value for -f
. It may reduce the memroy requirement.
Hi, does this week's release already implement the change?
Not yet. I will push it to the github HEAD this weekend.
Ok, super! Thanks a lot!
Hey there,
Sorry to bug you again 😅 Do you have any news on the memory consumption update?
Hi @chhylp123
Hope you are doing well and had a good start of the year! Do you by any chance have any updates on the memory consumption fix?
Hi again,
Apologies for opening a new issue but this time it is on a different topic.
We are trying to assemble a giant genome of ~30 Gb using 70x Hifi coverage and we have encountered a substantial memory consumption of circa ~2T. These are the flags we've used:
These are the resources used:
Notably, the first two error correction rounds are reasonable memory wise but the last one nearly doubles. Is this expected?
Some questions listed as points for brevity:
-k
and-w
raise memory consumption but given the speed boost and theoretical assembly benefits I would rather keep using them. I was wondering whether there is anything else I could tweak to reduce memory consumption without sacrificing assembly quality?-f
parameter?size of buffer
mean? Is this the size of the bloom filter?Thanks a lot once again!