klauspost / reedsolomon

Reed-Solomon Erasure Coding in Go
MIT License
1.83k stars 243 forks source link

Compatibility with other ReedSolomon Data #90

Closed racerxdl closed 5 years ago

racerxdl commented 5 years ago

I'm not sure how the sharding process work, but I have no idea what the shards really are.

I though they were the bytes itself, but if I ran against a RS(255,239) generated parity, it doesnt work.

How that library parameters correlate to normal RS Libraries for CCSDS, DVB-S and others?

Quick example: DVB-S uses RS(204,188). This is a example payload:

B8 1C 09 08 34 30 B8 A3 DA 60 29 C4 3B F7 6A EB F5 5F E9 BC 88 1B 8F 33 E2 EE E4 E0 9A 83 5F 0B C1 39 8D 92 2B 06 E0 B6 1A 4B EB 99 DC 54 C8 FB B4 1F B8 41 91 85 65 1F 5E 43 C5 88 9D 31 B1 54 EE 50 91 67 E0 7A 41 21 79 B2 EA 51 82 1A F3 A1 29 C4 F4 9A 3B 5C 9B CB A7 44 2C 67 E9 52 77 FD CF 91 5E 98 38 AF 6C 1C 97 B4 8E 44 DA 65 22 A1 30 39 5F 68 3C 8F 74 DC C5 35 61 40 B8 7C 6E F6 99 C8 AB 4C 04 57 E6 0F AB DE 07 3B ED 67 90 AE 9C 18 B7 AC 4E 16 5B 8A 26 C3 29 75 08 C1 CD 7B AF 1B E2 59 4D D1 AC E5 EA 5C 7D C9 0C B6 2B B4 8F 88 3B AC 53 79 0F B6 21 B4 C5 BA 3D A4 91 C0 B3 89 D0 2C 40 8D CB C0 05 5B 95 05

The last 16 bytes are the parity, first 188 are data.

darrenldl commented 5 years ago

This library works at the block level rather than byte level, so it's most likely not compatible with your use case. So instead of appending parity data to a single byte string, this library generates parity blocks to deal with missing blocks of data - you get a group of blocks rather than a new byte string at the end.

For your use case, something like this would be a better fit, but I have never used this particular library, so I can't say anything about quality or usability.

A shard is just a block of bytes.

racerxdl commented 5 years ago

Ahh great! I will check this RS from colin-davis. Looks like its what I'm looking for. So far I'm using a wrapper I did for libcorrect (https://github.com/racerxdl/gorrect/blob/master/ReedSolomon.go) but I wanted a pure go solution as well.

Thanks! I will check that.

Lucas