search

apple / foundationdb

FoundationDB - the open source, distributed, transactional key-value store
https://apple.github.io/foundationdb/
Apache License 2.0
14.53k stars 1.31k forks source link

Use homomorphic hash to checksum Replicas #3463

Open sfc-gh-mpilman opened 4 years ago

sfc-gh-mpilman commented 4 years ago

Adrian (from Snowflake) proposes the use of a homomorphic hash function open sourced by Facebook to verify the consistency of partitions within FDB.

Currently we use the consistency checker to achieve this. However, in the case of a corruption it could potentially take the consistency checker a very long time to detect it. Furthermore this could also be used to verify the correctness of a backup and/or snapshot backup without comparing the whole database.

The above document discusses more details.

jzhou77 commented 4 years ago

This looks very promising. Is there any benchmark number to show the CPU overhead of this hash?

sfc-gh-mpilman commented 4 years ago

There is a paper for this with (very few) numbers.