Open mr-eyes opened 7 months ago
Yep - and in fact, the whole _HashesWrapper
area of things is a great target for oxidation 🦀 , with potentially far-reaching speed and memory improvements!
This might be something that https://github.com/sourmash-bio/sourmash/pull/2943 will help with, or perhaps a targeted effort independently of that.
also kind of related: https://github.com/sourmash-bio/sourmash/issues/2898
(Yes, the relevant calculations are being moved into rust in #2943.)
This is an expected limitation in sourmash when working with signatures in Python data structures. Python dictionaries are restricted in the amount of data they can hold, regardless of available RAM.
Working on a super large signature like this can be rare to happen, but I am reporting the bug anyway.