search

VladRodionov / bigbase

BigBase - read optimized, fully HBase-compatible, NoSQL Data Store
GNU Affero General Public License v3.0
9 stars 1 forks source link

Koda: Add background recompression of objects in cache #30

Open VladRodionov opened 10 years ago

VladRodionov commented 10 years ago

Idea is to use the fastest compression when object is being placed into cache and later re-compress it using more advanced version (LZ4 =>LZ4-HC).

Some optimizations: https://groups.google.com/forum/#!topic/lz4c/fdlAfZzSAr4

VladRodionov commented 10 years ago

Done. Added StorageOptimizer threads, can configure # of threads and optimization level: 1-5

1 - It recompresses to LZ4HC all objects 2 - 30% (least recently used) to GZ, other to LZ4HC 3 - 50%-50% GZ/LZ4HC 4 - 75% to GZ 5 - 100% to GZ

VladRodionov commented 10 years ago

This is probably not for the first release, as since test StorageOptimizerTest fail when using put with codec method, but succeed when using regular put (see code).

VladRodionov commented 10 years ago

Another thing: this will work only for block cache, as since row cache uses API which has not been converted yet to multi-codec.