search

Tencent / Tendis

Tendis is a high-performance distributed storage system fully compatible with the Redis protocol.
http://tendis.cn
Other
2.87k stars 317 forks source link

Avoid reading non-meta keys to invoke compaction_filter during compaction #234

Open leonliao opened 1 year ago

leonliao commented 1 year ago

Description

Tendis saves data's TTL in the value of RT_DATA_META-type keys ONLY . And it utilizes the RocksDB's compaction_filter to read TTL out of value and then checks whether the key is expired.

So it is not necessary to read blob value of non RT_DATA_META-typed keys. Reading blob_values out of RocksDB consumes too much read-IO bandwidth.

Expected Behavior

Only reads the value of RT_DATA_META-type keys

Current Behavior

Reads all value of all type of keys.

Possible Solution

Use the FilterBlobByKey of RocksDB's blobdb feature, checks whether it is necessary to read blob value.

Context

The recommended solution saves the disk read IO bandwidth significantly when using set/hash with "big" values (>min_blob_size) when enabling RocksDB's enable_blob_files.

takenliu commented 5 months ago

tendis2.7.0版本参照leonliao同学的pr和建议，对此问题进行了优化，再次感谢～