Polkadot parachains not producing blocks until next session after v1.2 runtime enactement

When runtime v1.2 got enacted all protocols for async backing got enabled, that halted parachains until next session. Parachains started producing blocks again after the new session.

This is identical with the kusama incident reported here: https://github.com/paritytech/polkadot-sdk/issues/3058, which should have been fixed.

Root cause

This is exactly the same issue as https://github.com/paritytech/polkadot-sdk/issues/3058 and the problem happen again because the fix in https://github.com/paritytech/polkadot-sdk/pull/3063 was not entirely correct. Because v2::handle_deactivate_leaves https://github.com/paritytech/polkadot-sdk/blob/master/polkadot/node/network/statement-distribution/src/lib.rs#L344 gets called even for leafs that do not have prospective parachains enabled yet, we end up deleting the topology we saved for when leaves with async backing will come here https://github.com/paritytech/polkadot-sdk/blob/master/polkadot/node/network/statement-distribution/src/v2/mod.rs#L829.

So the moment async backing gets enabled, we end-up in situation where unused_topologies is empty, so without any saved topology this code will never get executed https://github.com/paritytech/polkadot-sdk/blob/master/polkadot/node/network/statement-distribution/src/v2/mod.rs#L634.

Hence why the next session fixed the issue.

TODO

[x] Root cause
[x] Write a proper postmortem.

paritytech / polkadot-sdk

Polkadot parachains not producing blocks until next session after v1.2 runtime enactement #4226

Root cause

TODO