search

paritytech / polkadot-sdk

The Parity Polkadot Blockchain SDK
https://polkadot.com/
1.94k stars 710 forks source link

availability-recovery optimizations #605

Open sandreim opened 1 year ago

sandreim commented 1 year ago

Related to https://github.com/paritytech/polkadot/issues/7411 . We should look further for optimizations for encoding/reconstruction to/from chunks. For example https://github.com/paritytech/reed-solomon-novelpoly/pull/2 or taking advantage of latest SIMD instruction sets.

burdges commented 1 year ago

cc paritytech/polkadot-sdk#598

sandreim commented 6 months ago

Adding https://github.com/AndersTrier/reed-solomon-simd to the options to explore

ordian commented 6 months ago

This will be a breaking change to the format, but it might be worth it - can be coordinated with node_features.

Also, while at it, we can also finally fix the merkle commitment to be a binary tree instead of complicated hexary trie. All of this is done in https://github.com/paritytech/erasure-coding and the benches show a decent gain in encoding as all: https://gist.github.com/ordian/0af2822e20bf905d53410a48dc122fd0

burdges commented 6 months ago

Ahh nice, it switches to reed-solomon-simd too. We observe a speed up like the one he claims? :)