search

jbapple / libfilter

High-speed Bloom filters and taffy filters for C, C++, and Java
Apache License 2.0
32 stars 6 forks source link

It is possible cache the bloom filter to NVMe? (support filters larger than 2^38 bytes) #4

Open hskun opened 2 years ago

hskun commented 2 years ago

libfilter is very fast. I have about 2^36 items to add in bf, it is possible cache the bloom filter to NVMe?

jbapple commented 2 years ago

You could use mmap in the C interface to allocate a libfilter_region's block pointer, I suppose. If you just accessed the NVMe space needed on each read or write, I suppose the OS might cache it anyway.

Thoughts?

hskun commented 2 years ago

You could use mmap in the C interface to allocate a libfilter_region's block pointer, I suppose. If you just accessed the NVMe space needed on each read or write, I suppose the OS might cache it anyway.

Thoughts?

I will have a try. Thank you!

jbapple commented 2 years ago

Looking more carefully at the code, I don't think it supports filters larger than 2^32 bytes. I'll take this as a feature request.

jbapple commented 2 years ago

When I run the following, the correct amount of memory is allocated on my machine:

BlockFilter::CreateWithNdvFpp(1ul << 34, 0.0001)

How much memory is allocated on your machine if you just call this factory method and then return from main? What tool are you using to measure the amount of memory allocated?