search

catid / leopard

Leopard-RS : O(N Log N) MDS Reed-Solomon Block Erasure Code for Large Data
BSD 3-Clause "New" or "Revised" License
141 stars 23 forks source link

Request: support >65536 pieces #2

Open Bulat-Ziganshin opened 7 years ago

Bulat-Ziganshin commented 7 years ago

65536 data+parity blocks limit is too low for some PAR3-style applications. I will be be glad to see it supported.

catid commented 7 years ago

Challenges:

(1) Error locator polynomial evaluation: Fast Walsh Transform on 2^32 words is not so fast? It would take 8.6 GB to represent 2^31 inputs...!

(2) Multiplication cannot be done via tables anymore, so it would get slower to calculate... Maybe Fp^2 would be better.

(3) The ErrorBitfield optimization class would be very important to implement but also very tricky.

(4) The current decoder wastes a lot of perf rounding up to the next power of two. I don't think that's going to fly when the data is this big!

(5) General memory handling issues...

catid commented 7 years ago

If applications require just slightly more than 2^16 blocks I could add e.g. 20 bit finite fields. I'd prefer to do that in response to a real demand rather than a precipitated need though.