search

rust-rse / reed-solomon-erasure

[Looking for new owners/maintainers, see #88] Rust implementation of Reed-Solomon erasure coding
MIT License
181 stars 61 forks source link

Moving this repo into an org? (Or alternatively add main contributors as collaborators for push access) #81

Open darrenldl opened 3 years ago

darrenldl commented 3 years ago

Tagging main contributors/users I have in mind (apologies if I missed any): @burdges @drskalman @rphmeier

To people above and also to (active) users and other contributors of this library: Sorry for my lack of responsiveness, life's been overwhelming a bit, and my line of work right now is not related to Rust. So it's difficult for me to keep track of Rust's side of things when I'm not actively using Rust.

To main contributors: I think it'd be better if I give push access/handover to you lot somehow on GH and crates.io

What do you guys reckon?

burdges commented 3 years ago

We're fine to handle PRs by other parties, and publishing on crates.io if you prefer. I could do so if nobody else wants to, but first..

We'll ideally migrate to an FFT design not only for encoding, but also for decoding using the formal derivative trick described in https://github.com/Bulat-Ziganshin/FastECC#faster and https://www.citi.sinica.edu.tw/papers/whc/3450-F.pdf We envision using the more flexible binary field additive FFT and formal derivative based approach described in https://www.citi.sinica.edu.tw/papers/whc/5524-F.pdf and implemented for high rate codes in https://github.com/SianJhengLin/Fast-algorithms-of-Reed-Solomon-erasure-codes We must understand the algorithm if we're going to make a low rate variant work though.

Those designs produce systemic codes, but I think the implementation here might not be systemic. It's therefore possible we should just fork the whole project and then harmonize the readme's to explain the situation.

Your two archiver crates look like the only production deployments on crates.io, well all other users being young blockchain projects. Aside from systemic v non-systemic, there exist different constraints in these new designs, like for high and low rate, so likely all variants should remain publicly accessible. Are your crate being used in a high rate or low rate way? Or both? We've this code in kusama and polkadot but its not yet being used until parachains launch in several months, so right now changes cause no problems for us.

darrenldl commented 3 years ago

We'll ideally migrate to an FFT design not only for encoding, ...

Ah right, so no point in fixing the caching if we are moving toward that.

Are your crate being used in a high rate or low rate way? Or both?

If I'm not minunderstanding the definitions, then both I think. But it's not an exceedingly important part of the formula since I don't know if there are active users of the archiver, on top of it being feature complete, and it can always stick to an older version of the crate if there are indeed issues.

I am definitely mostly thinking about other users, but I am not familiar with how involved their use with this crate is either. So if you don't think there would be a problem from lack of updates for now, then maybe I'm just overreacting (seeing issues being linked jolted my memory of this crate).

burdges commented 3 years ago

There is an error polynomial in an FFT decoder based upon the formal derivatives trick ala https://github.com/Bulat-Ziganshin/FastECC#faster which maybe still benefits from caching, but requires roughly the number of shards. It's actually why high rate decoders with fewer parity shards than data shards differ slightly from low rate decoder with more parity shards than data shards.

darrenldl commented 3 years ago

I think I'll just add everyone tagged in this issue to the collaborator list - you don't have to accept the invite, please feel free to reject if you don't feel like it.

I am definitely phasing out my involvement with Rust for the time being, and have a hard time keeping up with learning new things in my spare time.