search

VladRodionov / bigbase

BigBase - read optimized, fully HBase-compatible, NoSQL Data Store
GNU Affero General Public License v3.0
9 stars 1 forks source link

Block cache: improve external storage in RAM overhead #44

Closed VladRodionov closed 10 years ago

VladRodionov commented 10 years ago

Currently, its ~ 140 bytes per one block because we do not have efficient serializer for StorageHandle.

VladRodionov commented 10 years ago

Reduced overhead to ~ 84 bytes by implementing custom serialization to/from byte[].

VladRodionov commented 10 years ago

Will keep it open. We need reduce overhead more. The goal is to fit into 64 bytes

VladRodionov commented 10 years ago

I made some optimizations and currently, total size of a record is 73 bytes. Opened new ticket to shave 4 bytes more - Issue #48

VladRodionov commented 10 years ago

Opened one more ticket to further reduce overhead - Issue #49.

VladRodionov commented 10 years ago

Reduced RAM overhead of a one disk (external) cache entry to 64 bytes.

VladRodionov commented 10 years ago

64 bytes per one object in RAM. ~16M objects (data blocks) per 1GB of RAM. With avg size of compressed block of 32K => 500GB of addressable disk storage.