search

paritytech / polkadot-sdk

The Parity Polkadot Blockchain SDK
https://polkadot.network/
1.63k stars 571 forks source link

Elastic Scaling #1829

Open eskimor opened 9 months ago

eskimor commented 9 months ago

Eventually parachains should be able to occupy more than one core at once in a single block, which allows them to scale beyond a single core if their collators are powerful enough.

We decided to split up the work on elastic scaling into two/three phases. First phase is make it work for parachains which trust their collator set and don't share collations with untrusted machines. With this restriction it is possible to launch elastic/fixed factor scaling without any changes to the candidate receipt. Phase 2 will then actually make changes to the candidate receipt in order for the collator set to be untrusted again. (We put the core index in the candidate commitments, so it is not possible to push the same collation to multiple backing groups). Also phase 2 or potentially phase 3, we actually implement everything that is necessary on the cumulus side for elastic/on-demand scaling vs. fixed factor, where we just buy n cores all the time.

This ticket is about phase 1 and entails the following tasks:

### Tasks
- [ ] https://github.com/paritytech/polkadot-sdk/issues/3168
- [ ] https://github.com/paritytech/polkadot-sdk/issues/3128
- [ ] https://github.com/paritytech/polkadot-sdk/issues/3309
- [ ] https://github.com/paritytech/polkadot-sdk/issues/3129
- [ ] https://github.com/paritytech/polkadot-sdk/issues/3144
- [ ] https://github.com/paritytech/polkadot-sdk/pull/3573
- [ ] https://github.com/paritytech/polkadot-sdk/pull/3509
- [ ] https://github.com/paritytech/polkadot-sdk/issues/3775
- [ ] https://github.com/paritytech/polkadot-sdk/issues/3576
- [ ] https://github.com/paritytech/polkadot-sdk/issues/3776
- [ ] https://github.com/paritytech/polkadot-sdk/issues/3130
- [ ] https://github.com/paritytech/polkadot-sdk/issues/3202
- [ ] https://github.com/paritytech/polkadot-sdk/issues/3132
- [ ] https://github.com/paritytech/polkadot-sdk/issues/3837
- [ ] https://github.com/paritytech/polkadot-sdk/issues/3219
- [ ] https://github.com/paritytech/polkadot-sdk/issues/3541
- [ ] https://github.com/paritytech/polkadot-sdk/issues/3131
- [x] Check scheduler/coretime/.. that there are no restrictions on multiple cores with same para.
- [ ] https://github.com/paritytech/polkadot-sdk/issues/1797
- [ ] Fun and profit for phase one.
- [ ] https://github.com/paritytech/polkadot-sdk/issues/4467
- [ ] https://github.com/paritytech/polkadot-sdk/issues/4468
- [ ] https://github.com/paritytech/polkadot-sdk/issues/4547
Polkadot-Forum commented 8 months ago

This issue has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/scaling-a-polkadot-evm-parachain/4319/2

eskimor commented 6 months ago

Get core index into the candidate receipt, also useful here.

eskimor commented 5 months ago

https://github.com/paritytech/polkadot-sdk/issues/2984

bkchr commented 5 months ago

collator protocol: Add an additional (backwards compatibility) advertise collation message, which triggeres a different collation fetch request (requesting also the head data of the parent) -> Make use of that head data in candidate validation.

What does that mean?

alindima commented 5 months ago

collator protocol: Add an additional (backwards compatibility) advertise collation message, which triggeres a different collation fetch request (requesting also the head data of the parent) -> Make use of that head data in candidate validation.

What does that mean?

Assuming we have two validator groups, two cores for a parachain and we want to back two candidates in the same relay chain block, the second group needs to know the parent head data (which is the output of validation of the first candidate). Considering that candidate A hasn't yet been backed, the only source for this information on relay nodes would be the prospective-parachains subsystem (which records this data after getting a Seconded statement).

This latency of transmitting the seconded statement will partly prevent parallel execution of the two candidate validations. We can fix this by passing in the head data in the collation fetching response (it only contains the hash for now).

Polkadot-Forum commented 3 months ago

This issue has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/parity-tech-update-for-february/6630/1