Accordingly, we know that EC decoding for a 4K segment is about 25% faster using 384 data shards instead of 1024 data shard. In contrast if we are erasure coding 128KB of data or more, using 1024 data shard scheme performs better than all other options (excluding 512 data shard scheme but that is off the table because there is no clear way of distributing the data shards between validators evenly while maintaining availability with any subset of 384 honest live validators). This is expected as FFT has $n * log(n)$ complexity so it slows down more rapidly than $n$. However, because 384 is not a power of 2, the algorithm builds a polynomial of 512 degree but only uses 384 of its coefficients to store data shards (and waste the other coefficients). As such when we lose the advantage of higher parallerization due to size of the data, the 384 data shards scheme falls behind all other scheme we tested.
So it is clear that for bigger size data such as those used for reconstruction of work packages (10MB+) we should use the 1024 shards scheme.
The question is if the advantage of 384 scheme for segments of 4KB, worth having two different erasure coding schemes, or we should just use 1024 data shard scheme for both cases.
We'll bench mark the advantage using 384 data shard scheme for segments in the real scenario and decide about having a 384 data shard scheme for segment reconstruction.
Another possibility is to use 256 data shard scheme with 768 recovery (redundant) shards, which is more efficient than 384 scheme for 4KB but it uses slightly more storage (because of the 256 extra recovery shards) but it makes the segment available even if only 1/4 validators are honest and available.
These are the benches for encoding and decoding fixed size data packages of size
[4KB, 8KB, 32KB, 64KB, 128KB, 256KB, 512KB, 1024KB, 2048KB, 4096KB]
using
reed-solomon-simdlibrary using the following "data:recovery shards" scheme:Accordingly, we know that EC decoding for a 4K segment is about 25% faster using 384 data shards instead of 1024 data shard. In contrast if we are erasure coding 128KB of data or more, using 1024 data shard scheme performs better than all other options (excluding 512 data shard scheme but that is off the table because there is no clear way of distributing the data shards between validators evenly while maintaining availability with any subset of 384 honest live validators). This is expected as FFT has $n * log(n)$ complexity so it slows down more rapidly than $n$. However, because 384 is not a power of 2, the algorithm builds a polynomial of 512 degree but only uses 384 of its coefficients to store data shards (and waste the other coefficients). As such when we lose the advantage of higher parallerization due to size of the data, the 384 data shards scheme falls behind all other scheme we tested.
So it is clear that for bigger size data such as those used for reconstruction of work packages (10MB+) we should use the 1024 shards scheme.
The question is if the advantage of 384 scheme for segments of 4KB, worth having two different erasure coding schemes, or we should just use 1024 data shard scheme for both cases.
We'll bench mark the advantage using 384 data shard scheme for segments in the real scenario and decide about having a 384 data shard scheme for segment reconstruction.
Another possibility is to use 256 data shard scheme with 768 recovery (redundant) shards, which is more efficient than 384 scheme for 4KB but it uses slightly more storage (because of the 256 extra recovery shards) but it makes the segment available even if only 1/4 validators are honest and available.