Reed-Solomon erasure code is a powerful error correction algorithm that can help improve data reliability and recovery in distributed systems. By introducing this algorithm in Dgraph, we can ensure that data is not lost or corrupted even in the case of node failures, network issues, BitFlip, Burst Error, disk erros or power loss.
It can result in a higher disk space requirement.
Links to Discuss, RFC or previous Issues and PRs
No response
Links to examples and research
There are several open-source projects that use Reed-Solomon erasure code, such as Ceph and Hadoop. It is also a well-known and widely-used algorithm in the data storage and recovery industry.
Dgraph currently does not support Reed-Solomon erasure code, and there is no easy workaround to achieve the same level of reliability and recovery without it.
Solution proposal
Proposal:
Introduce Reed-Solomon erasure code as a built-in feature in Dgraph. This could be done by implementing the algorithm in the storage layer(Badger), and providing options for users to configure the level of redundancy and recovery they require.
Benefits:
Improved data reliability and recovery in the case of node failures or network issues
Increased confidence in the stability and resilience of the Dgraph system
No need to use HA all the time. The user can explore shards at max.
Use case
Reed-Solomon erasure code is a powerful error correction algorithm that can help improve data reliability and recovery in distributed systems. By introducing this algorithm in Dgraph, we can ensure that data is not lost or corrupted even in the case of node failures, network issues, BitFlip, Burst Error, disk erros or power loss.
It can result in a higher disk space requirement.
Links to Discuss, RFC or previous Issues and PRs
No response
Links to examples and research
There are several open-source projects that use Reed-Solomon erasure code, such as Ceph and Hadoop. It is also a well-known and widely-used algorithm in the data storage and recovery industry.
For windows https://github.com/Yutaka-Sawada/MultiPar
More about the algo https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_correction
A paper on the algo http://web.eecs.utk.edu/~jplank/plank/papers/CS-05-569.pdf
Reed Solomon Encoding - Computerphile https://www.youtube.com/watch?v=fBRMaEAFLE0
Current state
Dgraph currently does not support Reed-Solomon erasure code, and there is no easy workaround to achieve the same level of reliability and recovery without it.
Solution proposal
Proposal: Introduce Reed-Solomon erasure code as a built-in feature in Dgraph. This could be done by implementing the algorithm in the storage layer(Badger), and providing options for users to configure the level of redundancy and recovery they require.
Benefits:
Additional Information
No response