Can't write correction data to separate file?

[brenthuisman]

I want to confirm that the encoding process always writes it's own chucks, and can't use the original file (+a (set of) correction file(s)) as base from which recovery is possible. The readme contains a comparison to Parchive, which by nature works like this, but if zfec can work like that, I can't find it.

[exarkun]

If I understand this question correctly (which took me a while to do) ZFEC supports a k-of-n encoding where there are k input blocks - also called "primary" blocks - and n - k output blocks - also called "secondary blocks". Then, any combination of k primary or secondary blocks can be used to reproduce the original k input blocks.

So, I believe the answer is yes - the "correction data" (aka the "secondary blocks") can be separated from the original data (aka the "primary blocks").

[brenthuisman]

OK, thanks for the answer! I see the zfec/zunfec commands do not support this: passing in the original file (together with some secondary blocks) to zunfec raises raise CorruptedShareFilesError("Share files were corrupted -- share file %r said that m was %s but another share file previously said that m was %s" % (f.name, nm, m,)), I presume due to the fact the file is larger than a block size.

[exarkun]

Ah, indeed. I think for this to work you would have to slice the original file into blocks before feeding them in to zunfec. The slicing is simple: the first M bytes are the first block, the second M bytes are the second block, etc, where M is the size of a block for the given input and encoding parameters.

However, if you did this, it would mean you had the whole original file - and so you have all of the primary blocks so the secondary blocks are unnecessary. A more typical use would be to keep the primary blocks that come out of zfec and then, after losing some of those, use the remaining ones together with some secondary blocks as replacements.

I can imagine a use-case where you keep the whole original file and then use secondary blocks to repair it. I believe you could implement this with zfec but the current command-line tools don't quite work this way (they would have to know to re-slice the file into primary blocks and then compare them to the results of decoding using some secondaries - then maybe have some policy for how to decide which version wins when there are conflicting results). I don't know if this imagined use-case is similar to what you might want to do though.

[brenthuisman]

I can imagine a use-case where you keep the whole original file and then use secondary blocks to repair it.

That's exactly what I'm after :) Similar to how parchive functions.

How would conflicting results arise exactly?

Looking at help(zfec._fec.Decoder.decode) and it's use (e.g https://github.com/tahoe-lafs/zfec/blob/0c0692f5bba1aa07a11d643682704d621c58c22a/zfec/filefec.py#L299), looks like I'm going to have to chunk the input file (the k input blocks, of which some may be damaged), if I don't want to rewrite the C function, correct?

Is there a way to know if a repair is needed?

[exarkun]

looks like I'm going to have to chunk the input file (the k input blocks, of which some may be damaged), if I don't want to rewrite the C function, correct?

Yes, that seems about right.

Is there a way to know if a repair is needed?

ZFEC doesn't give you a strong way. If you have extra blocks around then you can see if decoding all combinations of the required set agrees. This can tell you if a block (or multiple) are corrupt but it can't tell you which and it can't tell you that no blocks are corrupt (some blocks might be corrupt in a way that exactly matches).

Tahoe-LAFS takes the hashes of data to be ZFEC encoded and then relies on the hashes, communicated out of band, to determine integrity of the data.