QA plan for `Nara`

[bedeho]

WIP: Will add link to past plan soon.

Remember to consider all apps, including CLI, where changes have been made.

┆Issue is synchronized with this Asana task by Unito

[chrlschwb]

Changes that need to be tested

• Council Election Cycle (Pioneer) https://github.com/Joystream/joystream/issues/4866
• Max number of subcategories (Pioneer) https://github.com/Joystream/joystream/issues/4852
• max workerNumberLimit for SWG/DWG (Pioneer) https://github.com/Joystream/joystream/issues/4822
• advanced-transactions (CLI) https://github.com/Joystream/joystream/issues/4903
• Content Curator Deletion power restriction (QN, CLI) https://github.com/Joystream/joystream/issues/4589
• FreezePallet Proposal (Pioneer) https://github.com/Joystream/pioneer/issues/4557

Timeline Oct 30 ~ Nov 6

[chrlschwb]

Outline

1. Stage an Ephesus network complete with RPC, QN, Pioneer and Atlas (freakstatic) 2. On the staging network test the following (ivant, polikosi)

RPC

• Are the following extrinsics available on polkadot-js? content:deleteChannelAsModerator content:deleteChannelAssetsAsModerator content:deleteVideoAsModerator content:deleteVideoAssetsAsModerator These extrinsics should be available. The functionality of the above extrinsics are out of scope

CLI

• Create a multisig account and test the following CLI commands to transfer 10 JOY from multisig to random account. constructTxCall advanced-transactions:constructUnsignedTxInitiateMs signUnsignedTx advanced-transactions:constructUnsignedTxApproveMs signUnsignedTx advanced-transactions:constructUnsignedTxFinalApproveMs signUnsignedTx

Atlas

• Confirm that CRT functionalities are working

3. Create a node with new runtime and expose RPC endpoint (freakstatic) 4. Perform a runtime upgrade from Ephesus to Nara using Pioneer Runtime Upgrade Proposal (ivant, polikosi) 5. Perform the following tests on Nara Network (ivant, polikosi)

RPC

• Test if polkadot-js api endpoint that was created in step 3 are working without error
• Check if following extrinsics are removed content:deleteChannelAsModerator content:deleteVideoAsModerator
• Confirm election stage durations from chain state

Pioneer

• Create a forum category with 10 sub categories
• Hire 50 workers into storage WG and distribution WG
• Pass the new proposal "pallet freezing"

CLI

• Create a multisig account and test the following CLI commands to transfer 10 JOY from multisig to random account. constructTxCall advanced-transactions:constructUnsignedTxInitiateMs signUnsignedTx advanced-transactions:constructUnsignedTxApproveMs signUnsignedTx advanced-transactions:constructUnsignedTxFinalApproveMs signUnsignedTx

Atlas

• Confirm that CRT functionalities are frozen

[bedeho]

Cover: https://github.com/Joystream/joystream/issues/4907

[bedeho]

Meta

Having a proper QA plan is critical now that we are on mainnet and need much greater assurance, and also when people involed are far less coordinated than when QA <=> dev relationship was much closer than today. Before mainnet, internal QA was done by mostly one person, and those plans may have reflected that.

A production level QA plan needs to be very specific about what the goals are, how success is determined and what resources are needed for a proper test. Without this, proper peer review of the plan is not going to be possible. Vague statements about goals will leave ambiguity, and there may be missed coverage of things that are critical. Also coordination and resource allocation within the QA team will be worse.

Feedback

This plan still can be significantly improved, it's still not clear to me what changes were made from the first version that received feedback, but for the benefit of next iteration please post a new entry into the thread with the new version.

1. Why are there two seemingly identical CLI sections?
2. Can you reorganize tests in terms of what they test, not what product is primarily used? So
   • CLI => "Multisig Functionality Still Works"
   • RPC => "Moderator Is Deletion Removed"
   • Pioneer => This is perhaps the bste example of how this does not work. Youare here grouping totally unrelated thigns, like pallet freezing and forum categories, despite them not directly having to do with each other, an reader not being able to easily understand that pallet freezing actually has to do with CRTs, not Pioneer per say. Pioneer is just a tool.
3. "Atlas: here you just Confirm that CRT functionalities are working". This is badly underspecified, what does this mean? What features exactly are you going to test, with what test scenarios? how many people are needed, what initial state is needed? Almost everything of relevance is missing. This is lastly not really a test of Atlas, its a test of at least two distinct things. That freezinga nd unfreezing works, and that CRT functionality works, and such test will likely span multiple products, no just Atlas.
4. It seems there is some formatting problems in this markdown, the overall structure is very confusing. Make sure that Markdown works for Github, since this is where display and reviw occurs, not in some other version of Markdown display.

[bwhm]

A few comments from me. In general, I would say it would be nice to add more context.

Some of this I shared on a call, and I'm sure you have though of this, but expanding/repeating it anyway as it's nice to include in the protocol. When testing that a new/changed/removed transaction behaves as intended after a runtime upgrade:

1. Ensure you know exactly how the transaction work before the runtime upgrade.
2. If applicable, ensure they would "behave" the same way regardless of whether the TX pointed interacts with "pre upgrade" state or "post upgrade" state.
3. List all the edge cases you want to test.

The last point is often overlapping with integration tests, but the point also applies to testing pioneer, atlas and the CLI.

--

1. Stage an Ephesus network complete with RPC, QN, Pioneer and Atlas (freakstatic)

If I were @freakstatic, I would have needed more context here. For testing purposes, we have typically deployed a playground-runtime first, which has only one validator, very different election and proposal periods, making testing easier. Furthermore, you will also start with a council elected, and leads hired for all roles. This is ideal for things that depends on council, proposals and working groups. It's also quick to redeploy and re-test if you happen to find any bugs.

If you haven't already, I would frequently start with a (playground) network with the "post-upgrade" runtime – in this case nara. Otherwise, you may spend time creating state on ephesus only to find you have to do it again, or you'll have to wait for some fixes on the CLI that could have been caught sooner.

For the final test, you may want to use a staging-runtime or a default one, but with some custom parameters (especially for elections and proposals).

2. On the staging network test the following (ivant, polikosi)

Again, lots of content should be added here. For each transaction:

• what state is needed to test each of these
• how and when do we achieve this
• which tools are we using
• what exactly are we testing
• what is the expected result before and after the upgrade
• etc.

For the multisig tests – before the upgrade:

• create 4 multisig addresses before the upgrade, with m/n 2/2, 2-off 2/3 and 4-off 3/5
• complete one transaction (initiate, approve and final approve) for each to ensure they work
  1. For the 2/2:
     • initiate a vesting.vest
     • initiate balance.transfer which depends on the vesting.vest to free the tokens,
     • approve the balance.transfer (not final approve)
  2. For the first 2/3:
     • initiate a balance.transfer
  3. For the second 2/3:
     • initiate a balance.transfer
     • approve the balance.transfer
  4. For the first 3/5
     • initiate a balance.transfer
  5. For the second 3/5:
     • initiate a balance.transfer
     • approve the balance.transfer (signer 2)
  6. For the third 3/5:
     • initiate a balance.transfer
     • approve the balance.transfer (signer 2)
     • approve the balance.transfer (signer 3)
  7. For the four 3/5:
     • initiate a balance.transfer
     • approve the balance.transfer (signer 2)
     • sign the final approve of the balance.transfer with (signer 3) just before the runtime upgrade, then broadcast a few blocks after the upgrade.

After the upgrade.

1. Final approve the vesting.vest, then the balance.transfer
2. Final approve
3. Final approve
4. Approve, then final approve
5. Approve, then final approve
6. Final approve

--

In order to test the content.x transactions (ref point 1. at the top), you need

• a council
• leads (at least storage, dist and content)
• multiple storage, and at least one dist, buckets which have to be enabled (but not actually deployed this purpose) *
• multiple content workers
• multiple channels and videos
• multiple curatorGroups...

* Probably not true on a playground-runtime

Then find out in which cases a curator can delete which channels/assets/videos before the upgrade – I don't remember exactly how does this works on ephesus, and I'm not sure how much help you'll get from the handbook...

[chrlschwb]

Requirements for the testing network

The usual playground-runtime wouldn't be ideal for this runtime QA testing because one of the things that need to be tested are the election parameters. Also, during 16th Oct meeting, it was decided to use a testnet with multiple validators for QA testing.

There are two separate required test networks

• Testnet No.1 A Nara testnet which has 3 validators, same parameters as nara . It should already have a council. polkadotjs-api, CLI, Pioneer, Storage node, distribution node, query node, Orion, Atlas instances required.
• Testnet No. 2 An Ephesus testnet with 1s blocktime and 3 validators. It should already have a council and constitutionality 1 for runtime upgrade proposal. Pioneer instance is required.

Scope of QA testing

Runtime upgrade from Ephesus to Nara

• This test will be done on Testnet No. 2
• Create wasm file for runtime upgrade to nara.
• Use Runtime Upgrade proposal in Pioneer to upgrade from Ephesus to Nara

Council Election Cycle

• Elect a council on Pioneer and confirm period lengths. This test will be performed on Testnet No.2 which is now in Nara after the Runtime upgrade from Ephesus to Nara test Idle period should be 56 hours, announcing period 16 hours, voting period 24 hours, revealing period 16 hours.
• Confirm election stage durations from polkadotjs-api This test will be performed on Testnet No.1 ChainState/Constants/council/idlePeriodDuration should be 201,600 ChainState/Constants/council/announcingPeriodDuration should be 57,600 ChainState/Constants/referendum/voteStageDuration should be 86,400 ChainState/Constants/referendum/revealStageDuration should be 57,600

Max number of subcategories

• This test will be done on Testnet No. 1
• Hire Forum WG lead
• Forum lead creates a forum category and 10 sub categories under that. Before the runtime upgrade, only 5 sub categories were possible.
• Forum lead creates 11 sub categories This should lead to failure.

max workerNumberLimit for SWG/DWG

• This test will be done on Testnet No. 1
• Hire SWG, DWG leads
• SWG/DWG leads hire 50 workers each. Before the runtime upgrade, the limit was 30 workers.
• SWG/DWG leads hire 51 workers This should lead to failure.

Content Curator Deletion power restriction

• This test will be done on Testnet No. 1
• Use polkadotjs-api to check if the following extrinsics are removed. Extrinsics/content/deleteChannelAsModerator Extrinsics/content/deleteVideoAsModerator Before the runtime upgrade, they were available.
• Check CLI to check if the following commands are removed. joystream-cli content:deleteChannelAsModerator joystream-cli content:deleteVideoAsModerator Before the runtime upgrade, they were available.

FreezePallet Proposal

• This test will be done on Testnet No. 1
• create a video channel and upload videos to Atlas.
• council passes CRT feature management proposal on Pioneer to turn off CRT features
• mint creator tokens, buy/sell tokens on market, initiating revenue splits and tokens staking on Atlas All CRT features should not work https://github.com/Joystream/atlas/issues/5360
• council passes CRT feature management proposal on Pioneer to turn on CRT features
• mint creator tokens, buy/sell tokens on market, initiating revenue splits and tokens staking on Atlas All CRT features should work

Multisig-related Advanced-transactions (removed from nara scope)

• This test will be done on Testnet No. 1
• Create a multisig account on polkadotjs-api
• Use Extrinsics/multisig/approveAsMulti to test multisig transactions.
• Use joystream-cli to test multisig transactions. advanced-transactions:constructTxCall advanced-transactions:constructUnsignedTxInitiateMs sign-offline:signUnsignedTx advanced-transactions:constructUnsignedTxApproveMs advanced-transactions:constructUnsignedTxFinalApproveMs

Any constructed call should be successfully signed and the transaction should be performed

[bedeho]

@chrlschwb

In Nara meeting today we concluded that the plan could have benefited from greater specificity about how the different initial settings of each testnet will be actually achieved, for example by stating which runtime profile will be used. Also, if chainspec builder is used, also which deployment option is being used, as this determines the initial genesis configuration to be used. Take this into account for future network QA plans.

[mnaamani]

@freakstatic Will add details of parameters chosen for the testnet(s) here

[ivanturlakov]

In progress...

Tested on https://65.21.109.54.nip.io/pioneer/#/council

Max number of subcategories

✅ Forum lead creates a forum category and 10 sub categories under that.
Before the runtime upgrade, only 5 sub categories were possible.

✅ Forum lead creates 11 sub categories.
This should lead to failure.

max workerNumberLimit for SWG/DWG

✅ SWG Limit

1. SWG - 25 workers(including lead)
2. Opening(hire limit 1) created - 29 applicants
3. Attempting to hire all 29 applicants --> Fail(MaxActiveWorkers) 54>50
4. Attempting to hire 26 applicants --> Fail(MaxActiveWorkers) 51>50
5. Attempting to hire 25 applicants --> Success --> Hired 25
6. SWG - 50 workers (including lead)

✅ DWG Limit

1. DWG - 26 workers(including lead)
2. Opening(hire limit 1) created - 26 applicants
3. Attempting to hire all 26 applicants --> Fail(MaxActiveWorkers) 52>50
4. Attempting to hire 25 applicants --> Fail(MaxActiveWorkers) 51>50
5. Attempting to hire 24 applicants --> Success --> Hired 24
6. DWG - 50 workers (including lead)

Content Curator Deletion power restriction

✅ Extrinsics are removed.
(Extrinsics/content/deleteChannelAsModerator, Extrinsics/content/deleteVideoAsModerator)


✅ CLI Commands are removed.
(joystream-cli content:deleteChannelAsModerator,
joystream-cli content:deleteVideoAsModerator)

Runtime upgrade from Ephesus to Nara

Tested on https://65.108.208.60.nip.io/pioneer/#/proposals/preview/2

Proposal Executed ⚠️ The application is unable to connect to the network Network status - connecting...

Council Election Cycle

(Elect a council on Pioneer and confirm period lengths. This test will be performed on Testnet No.2 which is now in Nara after the Runtime upgrade from Ephesus to Nara test)

Tested on https://65.108.208.60.nip.io/pioneer/#/council

⚠️ As described in the previous step, after the update the app fails to connect to the network, but there is an option to check the data in the Polkadot app

Target - 1s

✅ Idle period should be 56 hours = 201 600 blocks

⚠️ announcing period should be 16 hours = 57 600 blocks

⚠️ voting period should be 24 hours = 86 400 blocks

✅ revealing period should be 16 hours = 57 600 blocks

[chrlschwb]

QA result for Atlas CRT feature testing https://github.com/Joystream/atlas/issues/5483

[freakstatic]

@ivanturlakov the "types" errors was expected after the runtime upgrade, it's necessary to upgrade the joystream-node after the Nara runtime upgrade. I have done that, please try again to access: https://65.108.208.60.nip.io/pioneer/#/settings?network-config=https://65.108.208.60.nip.io/network/config.json

[freakstatic]

@ivanturlakov also the expected council duration parameters is here: https://github.com/Joystream/joystream/issues/4866#issuecomment-1732561544 please update your tests results based on that

[ivanturlakov]

Tested on https://65.108.208.60.nip.io/pioneer/#/settings?network-config=https://65.108.208.60.nip.io/network/config.json

@freakstatic thank you! Works as expected 👍

Council Election Cycle

(Elect a council on Pioneer and confirm period lengths. This test will be performed on Testnet No.2 which is now in Nara after the Runtime upgrade from Ephesus to Nara test)

✅ Idle Period: 14 days = 201 600 blocks ✅ Announcing Period: 6 days = 86 400 blocks ✅ Voting Period: 4 days = 57 600 blocks ✅ Revealing Period: 4 days = 57 600 blocks

target 1s

[ivanturlakov]

Tested on: Pioneer: https://65.21.109.54.nip.io/pioneer/#/proposals/past Atlas: https://65.21.109.54.nip.io/channel/

• There is no freezePallet option in the list of Proposals, so I used Polkadot.js
• Atlas contains some bugs(file upload error, links to channels and tokens - 404), which do not allow you to test all functions, I was able to test only the function of creating a new token and buy/sell

FreezePallet Proposal

freezePallet Proposal Executed

✅ Attempting to create a token > Fail ⚠️ Attempting to buy/sell tokens > Links to channels and tokens => 404

UnFreezePallet Proposal Executed

✅ Attempting to create a token > Success

switched to https://jsdao-nara-atlas.vercel.app/portfolio

Freeze again

✅ Attempting to create a token > Fail

⚠️ Attempting to buy tokens > Success https://polkadot.js.org/apps/?rpc=wss://65.21.109.54.nip.io/ws-rpc#/explorer/query/0xff4f43177bcfd8da57028859100bb087e863c89998be5f02e6a72b729e34f919

⚠️ Attempting to sell tokens > Success https://polkadot.js.org/apps/?rpc=wss://65.21.109.54.nip.io/ws-rpc#/explorer/query/0xed1271870a2e6d6e0c46454dda58db109153a1fe0a619f69794109a462927eb1

[ivanturlakov]

freezePallet Proposal view does not allow us to understand which exact action is proposed - ON/OFF would be good to show a widget

[ivanturlakov]

Tested on https://65.21.109.54.nip.io/pioneer/#/proposals/past

freezePallet Proposal Specific parameters

The toggle button is a bit confusing, maybe we should replace it with a:

• Text field - Current status Disabled
• Checkbox(default state=disabled, createProposal btn=disabled, label=Enable Pallet)
• Warning

Current flow

Current Status - Disabled I am trying to - Enable

Specific parameters default state - Disable, Create Proposal button is active(result will be the Proposal with the opposite state)

If I toggle to Enable

⚠️ Observation: no matter of toggle btn state result will be the Proposal with the opposite state, and always executes

[ivanturlakov]

FreezePallet Proposal retest

Atlas https://65.21.109.54.nip.io/ Dev Testnet

Pioneer https://65.21.109.54.nip.io/pioneer/#/settings wss://65.21.109.54.nip.io/ws-rpc

#	Enabled Pallet	Disabled Pallet
Create Channel	✅ Success	✅ Success
Create Token	✅ Success	✅ Fail
Buy Token	✅ Success	✅ Fail
Sell Token	✅ Success	⚠️ Token page error
Start Revenue Share	✅ Success	✅ Fail
Portfolio > Claim Your Share	⚠️ Button Inactive	⚠️ Button Inactive
Token Page > Stake/Claim Share	⚠️ Token page error	⚠️ Token page error

⚠️ Means there is no possibility to test ✅ Means expected result

Token page(if it was bought at least once) opens with an error