Nara upgrade checklist

[kdembler]

This document describes plan and a checklist of what needs to happen to properly execute Nara upgrade.

Before submitting proposal

• [x] Confirm no issues with wallets and sending tokens (@kdembler)
• [x] Confirm no expected issues with infra when upgrade happens (@freakstatic, @kdembler)
• [x] Confirm no issues in CRT pallet after code review (@bedeho)

First approval term

• [x] Calculate target block B for upgrade proposal execution, with expected UTC date T corresponding to B. For execution, we should target middle of revealing period of term X+1. This way we ensure that term X+1 doesn’t change duration mid-term and only term X+2 will use council term length from Nara. B = end_of(X) + 1 + 129,600 + 43,200 + (43,200 / 2). We may want to adjust to target specific part of revealing period so that upgrade executes during working hours - we're targeting block 6,374,350 ~= Monday 26/02 at 13 CET ~= 2024-02-26 12:00:00 UTC
• [x] Submit the upgrade proposal with trigger block B (@kdembler)
• [x] Confirm proposed runtime blob against fixed Nara runtime code shasum (DAO, @mnaamani, @freakstatic)
• [x] Review runtime code diff between fixed Nara runtime code shasum and master (@mnaamani, @kdembler)
• [x] Approve the upgrade proposal in term X (DAO)
• [x] Notify exchanges, wallet providers, Subscan, and validators that we will be upgrading at T. We don’t expect any action needed on their side. Confirm with Nova Wallet team that their metadata portal will be updated automatically. (@bedeho, @kdembler)
• [x] Start rolling upgrade of infrastructure (@mnaamani, @kdembler):
  1. Exclude 50% of RPC and QN backend servers from public load balancers.
  2. Upgrade those to Nara versions and start syncing (Nara QNs pointed at Nara RPCs).

Second approval term

• [x] Review DAO governance processes to make sure we’re prepared to term length change (DAO)
• [ ] Start upgrading private QNs of storage/distribution providers to Nara versions (DAO)
• [x] Approve the upgrade proposal in term X+1 (DAO)
• [x] At T - 1d, apply Nara changes to GitHub repo (@mnaamani):
  • Merge nara→master PR
  • Do Nara Github release with latest pre-built binaries
  • Publish latest packages (CLI, types, metadata) to NPM and mark as latest versions
• [x] Prepare PRs for merging nara branches into production for Atlas (Gleev) and Pioneer (JSG)
• [x] At T - 2h, put Atlas and Pioneer instances into maintenance mode (JSG, @kdembler)
• [x] At T - 30m, update public load balancers to point to Nara versions of QNs and RPCs. (@mnaamani , @kdembler)
• [x] At T - 25m, update status server with the Nara version pointed at updated public infra. (@DzhideX)
• [x] At T start confetti showers 🎉 (everybody)
• [x] At T + 5m, confirm everything works as expected and infra is working fine. If yes, stop the Ephesus QNs and RPCs and upgrade to Nara versions. (@mnaamani, @kdembler)
• [ ] At T + 5m, update https://metadata.joyutils.org (@kdembler)
• [ ] At T + 30m, merge Atlas (Gleev) and Pioneer Nara PRs so they start deploying to production. Confirm on localhost that those versions pointed at public infra work fine. (JSG)
• [ ] At T + 1h, re-enable access to Atlas and Pioneer instances. (JSG, @kdembler)

After upgrade

• [ ] Announce the upgrade and market the hell of it (JSG, DAO)
• [ ] Update Forkless Upgrades section in Handbook: https://handbook.joystream.org/system/blockchain#forkless-upgrades (DAO)

┆Issue is synchronized with this Asana task by Unito

[mochet]

Just one small thing to note is that any proposals in-flight will be cancelled at the time of the runtime upgrade.

[freakstatic]

@kdembler the CRT pallet will be unfrozen by default

looks pretty complete, good work 👍 should we add some upload tests on gleev and CRT features (like buy/sell tokens), after disabling maintenance mode, to make sure it's all working as expected?

[freakstatic]

Just one small thing to note is that any proposals in-flight will be cancelled at the time of the runtime upgrade.

For the runtime upgrade proposal it seems it goes to a "dormant" state after it get's the council approval

[mnaamani]

When we reference the "commit" for the runtime .. given that there could be other non-runtime related changes possibly, let use be more clear that it is the "runtime code shasum" produce by this script: https://github.com/Joystream/joystream/blob/nara/scripts/runtime-code-shasum.sh

[mochet]

Just one small thing to note is that any proposals in-flight will be cancelled at the time of the runtime upgrade.

For the runtime upgrade proposal it seems it goes to a "dormant" state after it get's the council approval

yes, that means it is waiting for the 2nd round of council voting, then it will go into grace period and after the grace period when it actually executes all other active proposals will be cancelled.

[kdembler]

@freakstatic Thanks, removed point about unfreezing @mnaamani Thanks, updated

[mnaamani]

Two jsgenesis operated nodes that I will probably also need to ensure are working are the "status" server and the "faucet" server.

[kdembler]

Updates after testnet upgrade:

1. We've discovered that the initial approach with timing the last approval in the 2nd round will not work, because the proposal in 2nd round also have an expiry block. Placing the last vote late enough for it to execute during revealing stage is not possible. Instead, we will use a "trigger block" functionality that lets us set exact execution block at proposal creation time. We have upgraded our testnet using this approach.
2. We have tested different versions of our software/nodes during an upgrade:
   1. Both Ephesus and Nara validator nodes were able to produce blocks before and after the upgrade.
   2. Both Ephesus and Nara QNs have continued to work after the upgrade. The Ephesus indexer experienced a crash as expected, but recovered and is processing blocks.
   3. ⚠️ The Ephesus QN processor has crashed once we have submitted the freeze pallet proposal as it's not recognized. This means that we should update all critical QN instances to Nara versions before the upgrade executes.
   4. Both Ephesus and Nara versions of Orion have survived the upgrade but the Ephesus Orion processor has crashed once we created a new channel.
   5. Seems both Ephesus and Nara faucets continue working fine after the upgrade. I think they both crashed during the upgrade but worked fine after restart.
   6. Storage Squid, Colossus and Argus continue to operate normally.
   7. Status server seems to have some small issues, @DzhideX will prepare an updated Nara version.
3. Bedeho has mentioned that his CRT pallet review efforts are not going as planned and that he's giving his green light and we should not wait for him with the upgrade.

Will upgrade the checklist to reflect those changes

[kdembler]

Update on wallet compatibilities:

1. OneKey - it doesn't have Joystream metadata at all, so it works the same regardless if it's Ephesus or Nara - all extrinsics except the balance transfer only support blind signing.
2. Polkadot Vault - will need manual metadata update via QR code.
3. Browser wallets - work fine, need metadata update to get extrinsic decoding. Also show balance from Nara RPC.

[freakstatic]

Updates after testnet upgrade:

1. We've discovered that the initial approach with timing the last approval in the 2nd round will not work, because the proposal in 2nd round also have an expiry block. Placing the last vote late enough for it to execute during revealing stage is not possible. Instead, we will use a "trigger block" functionality that lets us set exact execution block at proposal creation time. We have upgraded our testnet using this approach.

2. We have tested different versions of our software/nodes during an upgrade:

   1. Both Ephesus and Nara validator nodes were able to produce blocks before and after the upgrade.
   2. Both Ephesus and Nara QNs have continued to work after the upgrade. The Ephesus indexer experienced a crash as expected, but recovered and is processing blocks.
   3. ⚠️ The Ephesus QN processor has crashed once we have submitted the freeze pallet proposal as it's not recognized. This means that we should update all critical QN instances to Nara versions before the upgrade executes.
   4. Both Ephesus and Nara versions of Orion have survived the upgrade but the Ephesus Orion processor has crashed once we created a new channel.
   5. Seems both Ephesus and Nara faucets continue working fine after the upgrade. I think they both crashed during the upgrade but worked fine after restart.
   6. Storage Squid, Colossus and Argus continue to operate normally.
   7. Status server seems to have some small issues, @DzhideX will prepare an updated Nara version.
3. Bedeho has mentioned that his CRT pallet review efforts are not going as planned and that he's giving his green light and we should not wait for him with the upgrade.

Will upgrade the checklist to reflect those changes

Great work compiling this 👍 As you mention, both faucet's crashed with closed connection but restart them fixed the problem and they kept working fine.

Some conclusion:

• We can upgrade the RPC, QN and Orion that Jsgensis is running, before the runtime upgrade.
• We should/can recommend that people running QN (like SP and DP workers) upgrade there QN before the upgrade to avoid crashes, it doesn't seem urgent since the CRT events don't seem to crash the QN as @kdembler mention, only the freeze pallet proposal crashes it and I don't expect for this proposal to be submitted anything soon since it's a "emergency button" feature.
• Critical QN's need to upgrade since anyone can submit a freeze pallet proposal and, if this QN's are running the ephesus version, they will crash.
• Pioneer and Gleev will have to be updated after the runtime upgrade as expected.
• We really need to use the trigger block function to submit the runtime proposal.