klauspost / reedsolomon

Reed-Solomon Erasure Coding in Go
MIT License
1.88k stars 248 forks source link

updated simple-en/decoder to use shard headers. #270

Closed spikebike closed 9 months ago

spikebike commented 9 months ago

For the examples each shard gets a header for magic value, shard number, maxShard, size, and sha256 checksum.

Simple-encode adds the header and simple-decode checks these to help the user know if any shared has the wrong magic value, the wrong size, the wrong checksum (corrupted file), and the wrong order. By detecting these it's MUCH easier to decode since corrupted shards are automatically excluded. I wrote a test.sh (included) that corrupts shards (length, magic, checksum, and order) and decodes to show what happens.

I'd also suggest (this is NOT in the pull request) that enc.split and enc.Encode include an optional argument called header, which is prepended to each shard. Similarly enc.Decode should take an optional argument to skip the header (number of bytes) Thoughts?

Or maybe reedsolomon should not include file I/O, and just include some examples?

BTW, ioutil is deprecated: As of Go 1.16.

klauspost commented 9 months ago

These are simple implementation examples - not cli tools for end users.

I am not interested in making a file format, nor pretend this is a tool that should be used by anyone.

You are welcome to implement your own CLI tool - and you can probably make it a lot better.