basho / leveldb

Clone of http://code.google.com/p/leveldb/
BSD 3-Clause "New" or "Revised" License
408 stars 182 forks source link

non-compression of level 0 should be reevaluated [JIRA: RIAK-2792] #209

Open matthewvon opened 8 years ago

matthewvon commented 8 years ago

One of the earliest performance helpers (in 2012) was to never compress writes from memory to level 0. Data compression would be added later during compaction cycle from level 0 to level 1. This needs review. There is new threading and throttling code since then to help. There is the faster LZ4 compression code to help.

At issue is that uncompressed level 0 files can now be "moved" to higher levels and never experience a compaction cycle that would add the compression. There are certain workloads such as time series and Riak handoff that could make the lack of compression highly likely.