Closed hadiasghari closed 9 years ago
I suspect it is more to do with python's handling of memory than a bug, but I'm not positive and I plan to investigate down the line.
That said there are some python tricks to reduce memory handling and when I made the pure python implementation it was (is) a fairly naïve direct implementation of the C version with no pythonic tricks. My first goal was compatibility, which I think is mostly achieved (except for some potential bugs as you've seen); that is the 0.6.X series. My next goal is to make the pure python version more pythonic (this will be the 0.7.X series when I get to it). After that I plan on focusing on optimizations for both memory and performance with the goal of being as good as the C version, that will lead to a 1.0 release.
So I'll leave this issue open until that point. I can probably synthesize a large amount of prefixes to get some good divergent memory usages but if I need a real set I'll follow up with you.
Hi,
This is more of a question. I have a pretty large number of prefixes (half a million) to load. The C implementation of the library uses about 340MB on an x64 machine to load that; the Python implementation uses 1.8GB for the same data on the same machine!
Is this a mere limitation and inefficiency of Python memory handling versus C, or is it perhaps pointing to some bug? I can upload the prefix list if that helps.
Thanks!