google / leveldb

LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.
BSD 3-Clause "New" or "Revised" License
36.29k stars 7.8k forks source link

Write Amplification factor for small key-values with frequent writes to levelDB #1143

Open bhogi50 opened 1 year ago

bhogi50 commented 1 year ago

Hi Team,

I am facing a serious issue with write amplification. Consider, There are key-values that will write to levelDB for every 5 seconds. Those key-values are always batched into a write_batch object and then written to levelDB using leveldb_write() for every 5 sec. You can assume total size of key-values will be 500 bytes.

With leveldb ( sync = false ) , I am getting eMMC write count of 221 writes. where, If I write values directly into eMMC by maintaining some offsets for each value, I am getting eMMC write count of 70 writes.

The difference is Huge. Is there any way to optimize leveldb eMMC write count ? FYI, I tried all options available in Options.h file.