tomerfiliba-org / reedsolomon

⏳🛡 Pythonic universal errors-and-erasures Reed-Solomon codec to protect your data from errors and bitrot. Includes a future-proof zero-dependencies pure-python implementation 🔮 and an optional speed-optimized Cython/C extension 🚀
http://pypi.python.org/pypi/reedsolo
Other
351 stars 86 forks source link

Encode parity shards #65

Closed alplabin closed 1 year ago

alplabin commented 1 year ago

Hi there!

I'm currently exploring ways to achieve a similar result as the reed-solomon Go library in Python. In the Go library, the encoding process operates on a double array of bytes, which is then passed through the codeSomeShards function to create parity shards.

I'm wondering if it's possible to obtain a similar outcome using Python. Any suggestions or insights would be greatly appreciated!

lrq3000 commented 1 year ago

By reading the sourcecode you linked to, it appears the Go library is using a matrix Reed-Solomon approach, which is another way to implement the algo with some specificities but the advantage is that it is easily parallelizable. Unfortunately, our lib reedsolo does not implement the matrix approach but the original sequence-based approach (unofficial term I am making up, I am not aware how this is called, it's just not the matrix approach).

You need to look for a library that offers Vandermonde or Cauchy Reed-Solomon. You may also have a look at another python library called Galois which implements its own Reed-Solomon codec, though I didn't yet check how it is done internally, so I'm not sure it is matrix based.

If you don't care about parallelization, reedsolo should be compatible weth whatever the Go library generates since reedsolo is a universal RS codec, in the Readme there is a link to a code snippet to autodetect the RS parameters from an encoded message.

alplabin commented 1 year ago

I see, thank you for clarifying! Unfortunately, I specifically require a matrix Reed-Solomon approach for my project. I will explore alternative libraries that can fulfill this requirement.