search

rust-rse / reed-solomon-erasure

[Looking for new owners/maintainers, see #88] Rust implementation of Reed-Solomon erasure coding
MIT License
178 stars 62 forks source link

Looking for new owners/maintainers #88

Open darrenldl opened 2 years ago

darrenldl commented 2 years ago

Please reply here if you are interested in taking over maintenance of this package on github and crates.io, cheers.

nazar-pc commented 2 years ago

We use this library at the core of Subspace Network and I'm willing to help with maintenance here, but I don't have deep knowledge in EC algorithms used here, so might not be able to carefully review corresponding code changes if someone proposes optimization or something like that.

@rphmeier as a major contributor, maybe Parity can adopt this library?

burdges commented 2 years ago

There are many parity devs who afaik are welcome to maintain what they like, but..

We abandoned the direct approaches here as being too slow for polkadot, due to us aiming for over 1000 validators per relay chain. We instead ported C code that uses an asymptotically faster approach based upon AFFT and formal derivatives, which gave us like a 400x improvement.

It's live in kusama and polkadot now from https://github.com/drahnr/rs-ec-perf/ and a development version cleaning up the mathematical abstractions exists at https://github.com/w3f/rs-ec-perf/

This crate remains better if you need only very small numbers of shards. Our crate is better if you need large numbers of shards and can aim roughly for powers of two somehow (and do not mind learning more math).

nazar-pc commented 2 years ago

Oh, then we might want to use that too. We have 128 data and 128 parity shards, which isn't very small. Is it compatible with this library?

burdges commented 2 years ago

We made zero effort to be compatible, our interface remains unstable, and even our wire format could change with changing extension field optimizations, although polakdot demands a migration path.

I've referenced the relevant papers in https://hackmd.io/@rgbPIkIdTwSICPuAq67Jbw/r1B8eRC1u btw.

Just fyi, we always use a Byzantine >2/3rd honest assumption in polkadot. We must set the true recovery threshold t <= 2f+1 to be a power of two. We actually produce 2t > 3f+1 shards, so almost 25% more, also to be a power of two, but discard these extras. So our AFFTs run at different sizes in different directions. We inherited code for doing <50% thresholds, which works, but we never debugged to maintained it, and it maybe commented out.

darrenldl commented 2 years ago

Small update:

Just realised I forgot to change member role in github org: everyone should be a owner now.

Right now we have @nazar-pc and @mvines who are owners of both the repo/org and the crate (also sent an invite to @burdges). Please feel free to make any modification and publishing as you see fit.

I've also added @nazar-pc and @mvines to rust-rse on npm should you want to publish the wasm build there as well (can't seem to find @burdges on npm).

Thanks for the help everyone!

nazar-pc commented 2 months ago

@irfanshaik11 it would be best to start with contributions. With all respect, we probably wouldn't want to make someone who never contributed to the repository and has a closed GitHub profile a maintainer of the project.

burdges commented 2 months ago

We're hopefully migratingf rom our own crate to https://github.com/AndersTrier/reed-solomon-simd btw