Erasure Coding

[DavidVorick]

Need to be able to split files up and store them in multiple places, and perhaps have some sort of file health indicator or whatever. Files need to go up and be healthy.

I don't currently know if the regenerating codes are finished/ready, so we should probably just use the longhair library again (if we can manage cross-compilation) or some slower library that has been written in go.

It's not a big deal to require the user to have the file to do repairs, especially since it's only a temporary requirement thanks to regenerating codes.

[lukechampine]

My vote is to use a Go library if possible. Efficiency shouldn't be a primary concern while we're in beta.

[DavidVorick]

golang libraries: https://godoc.org/code.google.com/p/rsc/gf256 https://codereview.appspot.com/3295042/

research papers: http://www.eecs.berkeley.edu/~rashmikv/research.html

For now if we can figure out how to get gf256 working that's my vote. Once we want higher efficiency and better stuffs we can email rashmik & co. and see if they have reference implementations of their new research.

[lukechampine]

Yeah I saw gf256 too. If rsc (Russ Cox) made it, then I trust that it's very well written. (he works in Kendall Square btw, we should say hi :P)

[DavidVorick]

rsc = reed solomon coding?

oh lol I guess not. welp that's misleading

[lukechampine]

That's a pretty funny coincidence haha

[kroese]

On Usenet its very normal to use PAR2 files for this, so I am not sure if this should be in Sia itself, since it would be the same as adding automatic zip-support to Sia. It's an responsibility of the client to add PAR2 files (or not) for erasure codes, or wether to use ZIP on their data or not, etc.

http://www.quickpar.org.uk/VerifyingAndRepairing.htm http://parchive.sourceforge.net/docs/specifications/parity-volume-spec/article-spec.html