codex-storage / nim-codex

Decentralized Durability Engine
Apache License 2.0
62 stars 22 forks source link

Generalize reverse index mapping in EC strategies #836

Open gmega opened 2 months ago

gmega commented 2 months ago

Although our erasure coding module would appear to accept any interleaving strategy (i.e. an arbitrary permutation) for input blocks, the actual code that performs the reverse mapping of block indices into positions in the encoding array is specific to the stepped strategy.

This means that, currently, attempting to use any other strategy, including the linear strategy, will result in breakage. Indeed, the reverse mapping $f(i, s)$ for an index $i$ under the linear strategy using $s$ encoding steps should be:

$$ f(i,s) = \left\lfloor \frac{i}{s} \right\rfloor $$

In general, reverse mapping should be a property of the strategy itself, and the erasure coding module should contain no code that performs it.

Other pieces with hard-coded index mapping

cskiraly commented 2 months ago

Agree, please use the generalized version. I would also make interleaving explicit in the parameters, as in my https://github.com/codex-storage/nim-codex/tree/encoding-multidim branch. That should have all the necessary code changed.