search

prashnts / pybloomfiltermmap3

Fast Python Bloom Filter using Mmap
https://github.com/prashnts/pybloomfiltermmap3
MIT License
130 stars 24 forks source link

Saving/Loading/Checking in a bloom-filter 'bytes' objects doesn't work #48

Closed pianist-coder closed 6 months ago

pianist-coder commented 2 years ago

I have code that creates a filter containing a bunch of bytes objects. While that Python interpreter is still running, I can load the saved filter into a different filter object and it works. However, if I quit the interpreter and load the filter again, it no longer works.

prashnts commented 2 years ago

When you say "objects", is it one of the primitives or something else?

Can you provide an example please? I'm not sure I can reproduce it.

pianist-coder commented 2 years ago

When you say "objects", is it one of the primitives or something else?

Can you provide an example please? I'm not sure I can reproduce it.

b'\x04y\xbef~\xf9\xdc\xbb\xacU\xa0b\x95\xce\x87\x0b\x07\x02\x9b\xfc\xdb-\xce(\xd9Y\xf2\x81[\x16\xf8\x17\x98H:\xdaw&\xa3\xc4e]\xa4\xfb\xfc\x0e\x11\x08\xa8\xfd\x17\xb4H\xa6\x85T\x19\x9cG\xd0\x8f\xfb\x10\xd4\xb8'

In this case is random 65 bytes

And tell please, we may use pybloomfilter3 from .so lib without install? And.. This not working in windows OS?

And another question, does the final size of the created filter depend directly on the size of the available RAM? I have 8 GB of ddr4, at most I can add 200 million items. And a person who has, say, 64 GB of ddr can create a filter with 2 billion elements.

Thanks very lot.

pianist-coder commented 2 years ago

Any updates?((

pianist-coder commented 6 months ago

Two years have passed. Do you fix the problem somehow or not?

prashnts commented 6 months ago

Hi, windows isn't supported. Maybe the recently released pickle feature could be what you need. Unfortunately, unless you're experiencing a bug we don't plan to implement new features.

pianist-coder commented 6 months ago

OK, thanks. Can't you save it to a pickle file anyway? To upload it to the program later?

prashnts commented 6 months ago

There's the concept about an object being picklable, basically a protocol that determines how to serialize an object to load at a later time. Support for this was implemented and then released recently. I will close this issue for now as I believe the issue is resolved, but you can reopen it.