Network scalability: 300 paravalidators and 70 parachains

[sandreim]

Parachain consensus is restricted to a subset of the relay chain validators via themaxValidators configuration parameter. The plan is to remove the parameter, but until then we will gradually increase it's value as we improve the protocol and performance characteristics of the implementation. maxValidators is bumped to higher values only if key performance metrics, specifically parachain block times and finality lag do not degrade when more validators participate in parachain consensus.

After improving our tooling for profiling, gathering metrics and debugging we identified bottlenecks which we are currently addressing.

Experiments and potential improvements are tested on Versi before enabling them on value bearing chains like Kusama and Polkadot. After a battery of tests successfully ran on Versi we have already bumped the number of paravalidators from 200 to 250 on Kusama and we are currently monitoring the key metrics.

The scalability project board is available here: https://github.com/orgs/paritytech/projects/63

Plan

• [x] https://github.com/paritytech/polkadot/issues/5962
• [x] https://github.com/paritytech/polkadot/pull/6401
• [x] https://github.com/paritytech/polkadot/issues/6832
• [x] https://github.com/paritytech/polkadot/issues/6904
• [x] https://github.com/paritytech/polkadot/pull/6905
• [x] https://github.com/paritytech/polkadot/pull/7038
• [x] https://github.com/paritytech/polkadot-sdk/issues/640

Q2 milestones:

• [x] Increase the number of parachain validators from 200 to 250 (supporting up to 70 parachains) on Kusama
• [ ] Increase the number of parachain validators from 250 to 300 (supporting up to 70 parachains) on Kusama
• [ ] Increase the number of parachain validators from 200 to 300 (supporting up 70 parachains) on Polkadot