speedb-io / speedb

A RocksDB compliant high performance scalable embedded key-value store
https://www.speedb.io/
Apache License 2.0
912 stars 71 forks source link

Paired Block Bloom Filter Algorith - BPK Adjustment to avoid performance degradation with bpk<=10 #161

Closed udi-speedb closed 1 year ago

udi-speedb commented 2 years ago

During @noamhaham's performance runs, he discovered that there is a performance degradation with the paired filter with bpk=10. suggests to adjust the user's bpk (round up) to end up in an effective bpk that is larger that the user's. This adjustment implies more memory consumption for the filters (relative to not rounding up). The more keys a filter has, the smaller the adjustment / memory-consumption-increase and vice versa => Noam expects this to be felt more in Level 0 and less as the level progresses. However, since most of the data is in the lower levels, he expects the total consumption increase to be negligible.

udi-speedb commented 2 years ago

Pull request: https://github.com/speedb-io/speedb/pull/163

erez-speedb commented 1 year ago

Performance needs to cover a few tests. Checking with 20 BPK and compare to previous version, make sure no degradation. Test on 8 BPK and compare to native bloom.

erez-speedb commented 1 year ago

Test on the 20 BPS show some degradation,

Image

The second test was OK

Image

erez-speedb commented 1 year ago

The 8 BPS test

Image

erez-speedb commented 1 year ago

Pass performance tests

udi-speedb commented 1 year ago

@erez says he has checked a branch named pared_bloom_filter-improved I will take a look at it now