ydb-platform / nbs

Network Block Store
Apache License 2.0
50 stars 14 forks source link

[NBS] Implement blob compression #9

Open qkrorlqr opened 7 months ago

qkrorlqr commented 7 months ago

The following sensors {project="nbs", cluster="mycluster", service="service", host="cluster", sensor="*ompressedBytesWritten", type="ssd"} observed on our clusters show that the data that our users store can be compressed very well (x2.5 - x3 with lz4 codec). We can save a lot of space if we store compressed blobs in blobstorage. But a naive implementation - simply compressing whole blobs - will not work - if we compress a 4MiB blob and store the compressed form we will need to read and decompress the whole blob when the user decides to read a single 4KiB block from it. That's why we will need to do something like this:

Chunk size and min compression ratio should be configurable via TStorageServiceConfig. In the future more complex logic may be implemented: e.g. we can track read request sizes and dynamically change chunk size and min compression ratio.

SvartMetal commented 3 months ago

Also, we should add CompressedBytes/UncompressedBytes metrics for Filestore to understand whether we need blob compression for Filestore or not