search

valkey-io / valkey-bloom

Rust based Valkey Module which provides a BloomFilter data type / APIs
9 stars 8 forks source link

add bf.load and bf.dump to support bloom filter aofrewrite. #17

Open wuranxx opened 1 week ago

wuranxx commented 1 week ago

this pr is for #8 https://github.com/valkey-io/valkey-bloom/issues/8.

design proposal

I use serde and bincode to deserialize and deserialize BloomFilterType.

bf.load not support override old key.

performance

bf.dump

I use memtier_benchmark to test 0.01 error rate and 1000000/10000000/10000000 capacity bloom filter bf.dump performance in my x86_64 linux pc. (use valkey 7.2.5)

bf.reserve 1_m 0.01 1000000
bf.reserve 10_m 0.01 10000000
bf.reserve 100_m 0.01 100000000
memtier_benchmark -s {ip} -p {port} -t 2 -c 10 -n 1000 --command='bf.dump 1_m'

2         Threads
10        Connections per thread
1000      Requests per client

ALL STATS
===================================================================================================
Type          Ops/sec    Avg. Latency     p50 Latency     p99 Latency   p99.9 Latency       KB/sec 
---------------------------------------------------------------------------------------------------
Bf.dumps      1182.57        15.86025        16.06300        21.88700        26.75100   1383940.59 
Totals        1182.57        15.86025        16.06300        21.88700        26.75100   1383940.59 

memtier_benchmark -s {ip} -p {port} -t 2 -c 10 -n 10 --command='bf.dump 10_m'

[RUN #1 100%,   3 secs]  0 threads:         200 ops,      59 (avg:      51) ops/sec, 680.89MB/sec (avg: 587.32MB/sec), 362.31 (avg: 388.40) msec latency

2         Threads
10        Connections per thread
10        Requests per client

ALL STATS
===================================================================================================
Type          Ops/sec    Avg. Latency     p50 Latency     p99 Latency   p99.9 Latency       KB/sec 
---------------------------------------------------------------------------------------------------
Bf.dumps        67.64       388.35744       335.87100       716.79900       737.27900    791423.82 
Totals          67.64       388.35744       335.87100       716.79900       737.27900    791423.82 

memtier_benchmark -s {ip} -p {port} -t 2 -c 10 -n 10 --command='bf.dump 100_m'

[RUN #1 100%,  43 secs]  0 threads:         200 ops,       6 (avg:       4) ops/sec, 742.53MB/sec (avg: 525.07MB/sec), 3882.82 (avg: 4338.45) msec latency

2         Threads
10        Connections per thread
10        Requests per client

ALL STATS
===================================================================================================
Type          Ops/sec    Avg. Latency     p50 Latency     p99 Latency   p99.9 Latency       KB/sec 
---------------------------------------------------------------------------------------------------
Bf.dumps         4.80       989.18400       991.23100       991.23100       991.23100    561368.64 
Totals           4.80       989.18400       991.23100       991.23100       991.23100    561368.64

three key dump size:

-rw------- 1 root root 119813429 Oct 19 18:28 100_m_dump
-rw------- 1 root root  11981521 Oct 19 18:29 10_m_dump
-rw------- 1 root root   1198333 Oct 19 18:29 1_m_dump

aof size:

total 129884
-rw------- 1 2000 2000 132993423 Oct 19 18:37 appendonly.aof.2.base.aof
-rw------- 1 2000 2000         0 Oct 19 18:37 appendonly.aof.2.incr.aof
-rw------- 1 2000 2000        88 Oct 19 18:37 appendonly.aof.manifest

redis bloom aof size(v2.4.2)

total 149432
-rw------- 1 2000 2000 153011140 Oct 19 18:35 appendonly.aof.2.base.aof
-rw------- 1 2000 2000         0 Oct 19 18:35 appendonly.aof.2.incr.aof
-rw------- 1 2000 2000        88 Oct 19 18:35 appendonly.aof.manifest

if need more test, please tell me.

wuranxx commented 5 days ago

@KarthikSubbarao Please help review this PR.

wuranxx commented 1 day ago

@KarthikSubbarao The review comments have been modified and replied. Please review them again.