search

buildbarn / bb-storage

Storage daemon, capable of storing data for the Remote Execution protocol
Apache License 2.0
137 stars 91 forks source link

Add new 'quorum' blob access #213

Open anguslees opened 4 weeks ago

anguslees commented 4 weeks ago

Add a new 'quorum' blob access, that provides high availability on top of already-durable storage.

Quorum blob access only requires that a subset of backends are available in order to function successfully. The exact quorum size is configurable, but almost all cases should use "smallest integer greater than half" for both read and write quorum size. ie: 2 out of 3 backends for single-failure tolerance, or 3 out of 5 backends for double-failure tolerance.

Writes (Put) must succeed on at least write_quorum number of backends, and occur in parallel. Reads (Get) must see at least read_quorum number of not-found responses before concluding the blob does not exist, and occur sequentially. FindMissing reads are performed in parallel on read_quorum number of backends, and results are merged.

Note: blobs are not replicated again after the initial Put, so the underlying storage should be durable.

EdSchouten commented 4 weeks ago

Note: blobs are not replicated again after the initial Put, so the underlying storage should be durable.

I don't think that's a very desirable property. What should users do if the underlying storage nodes do get out of sync (e.g., due to disks in a server getting replaced)? Do we provide any tools to do the copying?