hbulgarini commented 7 months ago

Paseo Testnet Runtime Upgrade Checklist

Preparation Tasks

[x] Upgrade Proposal Review: Confirm that the core team agreed on the upcoming Runtime Upgrade.
[x] Impact Analysis Completed: Which would be the impact of this Runtime Upgrade?.
[x] PR Creation:: Run the script to backport the changes from the Polkadot Fellowship Runtime and solve all conflicts.
[ ] Benchmarking: Run benchmarks according to Paseo hardware specs and upload them as a different PR targeting the previous PR mentioned.
[ ] Chopsticks testing: Fork Paseo relay-chain and its system chains and commit the targeted runtime upgrade.
- [ ] Epoch and Era changes: Make chopticks to produce the necessary amount of changes to change the Era and assure blocks are still produced.
- [ ] XCM transfers: Make a teleport to the Paseo assethub and PAS reserved transfer to any available parachain.
[x] Try-Runtime: Run try-runtime and mke sure that no state inconsistences are reported.
[ ] Polkadot Fellowship contact: In case of any issues detected during testing, contact any Polkadot Fellowship member to address the concern. If no issues this can just be checked.
[ ] Runtime upgrade on test instance: Commit the Runtime upgrade on the Paseo test instance and leave it running for a whole day.

Execution

[ ] DB Backup: Ask any infrastructure provider to produce a backup of the network before commiting the upgrade.
[x] Notify Stakeholders: Distribute the communication plan through all relevant channels.
[x] Schedule Upgrade Window: Decide on and communicate the upgrade window to all stakeholders. Use always CET Timezone for scheduling or communicating deadlines.
[ ] Final Pre-Upgrade Check: Conduct a last-minute check to ensure all systems and stakeholders are ready.
[ ] Commit Upgrade: Initiate the runtime upgrade process according to the planned procedure. Use always CET Timezone for scheduling or communicating deadlines.
[ ] Monitor Deployment: Closely monitor the network for any immediate issues during the upgrade.

Validation

[ ] Verify Network Stability: Check that the network returns to normal operation post-upgrade.
[ ] Confirm Upgrade Success: Validate that the new runtime version is correctly implemented across the network.
[ ] Stakeholder Feedback: Collect and review feedback from validators, developers, and users.

Post-Upgrade

[ ] Announce Completion: Inform stakeholders that the upgrade has been successfully completed.
[ ] Post-Upgrade Review: Conduct a review meeting to discuss the upgrade process, identify any issues, and document lessons learned.

Troubleshooting

[ ] Issue Tracking: Log any issues encountered during the upgrade in a dedicated tracker.
[ ] Resolve Critical Issues: Prioritize and address any critical issues that arise.
[ ] Communicate Ongoing Issues: Keep stakeholders informed about any unresolved issues and the plans for resolution.

hbulgarini commented 7 months ago

We see a considerable number of migrations in this runtime upgrade and the state trie migration took our attention so we will need someone of the fellowship to address the weights values according to Paseo hardware.

educlerici-zondax commented 7 months ago

Runtime version 1.2.0 is to be enacted this coming 2nd of May, Thursday. ⏫

As you might already seen the enactment of this upgrade caused some downtime on parachains - more context on this amazing postmortem.

We have already notified the infrastructure maintainers of Paseo to run the patched version before the upgrade date. Although, the team will choose the proper timing for the upgrade so it happens in a safe time frame, close to a session change, minimizing the potential downtime if any.

We will keep you all posted.

Thanks!

al3mart commented 7 months ago

Results of try-runtime run at: https://gist.github.com/al3mart/121d88d4bb5175c107d9adc0f3cbaba1

paseo-network / runtimes

Runtime Upgrade: 1.2.0 #71