[NBS] Ability to save block checksums in TBlobMeta to check them upon ReadBlob

ydb-platform / nbs

Network Block & File Store

Apache License 2.0

58 stars 22 forks source link

[NBS] Ability to save block checksums in TBlobMeta to check them upon ReadBlob #196

Closed qkrorlqr closed 1 month ago

qkrorlqr commented 9 months ago

In case of data corruption bugs we need to be able to track the layer which causes the issue. This issue is about recording block checksums after WriteBlob in TBlobMeta (https://github.com/ydb-platform/nbs/blob/dd8b1e2aee138b3d6b027967c7ccbbed1a0fb722/cloud/blockstore/libs/storage/protos/part.proto#L148) to check that the data read via ReadBlob is the same data that was written upon WriteBlob. Since it implies storing a lot of metadata it will cause a performance hit => we can't enable this logic by default, need to be able to enable it via features config.

qkrorlqr commented 9 months ago

need to be able to enable it via features config.

there is actually a lot of value in enabling this by default but I will limit the block range which is affected and this block range will be configurable via TStorageServiceConfig

qkrorlqr commented 9 months ago

Basic implementation done. TODO:

implement checksums for unconfirmed blobs
implement checksums for patched blobs

qkrorlqr commented 6 months ago

implement checksums for unconfirmed blobs

done

qkrorlqr commented 6 months ago

the only thing that's left to do is:

implement checksums for patched blobs

everything else is implemented, tested, deployed and seems to work fine

qkrorlqr commented 1 month ago

patching-related thing moved here: https://github.com/ydb-platform/nbs/issues/1986