attic-labs / noms

The versioned, forkable, syncable database
Apache License 2.0
7.44k stars 266 forks source link

Idea: maybe chunks.Chunk should be compressed, rather than uncompressed #3754

Open ghost opened 7 years ago

ghost commented 7 years ago

Now that (most) values are just byte slices, Decode and Encode don't make a ton of sense. If the chunk.Chunk abstraction held compressed bytes (and nbs stopped doing compression internally), it would allow for

-NBS to do a better job of limiting memTables to compressed size. -Help sync avoid re-compressing chunks it just decompress to walk refs -Allow ValueStore to do a better job of buffering values (i.e. it can eagerly compress leaf values to chunks -- as it never needs to walkRefs on them).