A Tool for Scrutinizing OpenGov Proposals (Don't Trust, Verify)

[liamaharon]

Background

Last week, Kusama Referendum 297 upgrading Relay and System chains to V1 runtimes failed due to a misconfiguration in the call and needed to be re-submitted.

There is clearly a need for a tool to ease the process for both the Fellowship and wider community to collectively verify the integrity of proposals prior to voting.

Existing work

https://github.com/AcalaNetwork/chopsticks is well suited for dry-running proposals and verifying they do as intended. @xlc has made the most progress with this, and has provided an example script of how to

1. Whitelist a Kusama runtime upgrade call
2. Dry-run the call, and inspect post-upgrade state using Polkadot.js

Kusama:

npx @acala-network/chopsticks@0.9.0 xcm --relaychain kusama --parachain statemine --parachain kusama-bridge-hub

ws://127.0.0.1:8000 for asset hub, ws://127.0.0.1:8001 for bridge hub, ws://127.0.0.1:8002 for kusama

Open https://polkadot.js.org/apps/?rpc=ws%3A%2F%2F127.0.0.1%3A8002#/js and run this JS

const number = (await api.rpc.chain.getHeader()).number.toNumber()
await api.rpc('dev_setStorage', {
 scheduler: {
   agenda: [
     [
       [number + 1], [
         {
           call: {
             Inline: '0x2c0042d9642af11541927831b1795a1d47e29b2491ce8166d65076e4d63d83ccf973'
           },
           origin: {
             system: 'Root'
           }
         }
       ]
     ],
     [
       [number + 2], [
         {
           call: {
             Lookup: {
               hash: '0xd51bacd5ea827cf48fd539d4cdd3edfc11de191ef7e3df301cebcb0cafccb5ab',
               len: 1423658
             }
           },
           origin: {
             system: 'Root'
           }
         }
       ]
     ]
   ]
 }
})
await api.rpc('dev_newBlock', { count: 2})

Specification (DRAFT)

A CLI that dry-runs and verifies arbitrary proposals, including those that involve XCM.

• Inputs
  • Some set of calls to dry-run (e.g. whitelist + execute referendum)
  • Expected state changes (e.g. :CODE is updated and nothing else)
• Outputs
  • Expected state changes and unexpected state changes
  • Exposes local RPCs allowing devs to connect to and play with post-upgrade state using Polkadot.js
• If convenient, consider dedicated subcommands to ease the process of verifying common types of proposals like runtime upgrades. Do not sacrifice the flexibility of the tool to this end.
• Likely will heavily lean on Chopsticks

Next steps

• I would love some feedback from @xlc on this issue and any details we can add to the specification, given his expertise with Chopsticks.
• It has been suggested we create a bounty for the community to create this
  • Need to clear up details around how much it will be, where to get the funding, and how to take applications (can someone connect me with someone experienced with this?)
  • I suggest we set a deadline where if there is no progress on the bounty in N months we (Fellowsihp) develop it ourselves, or I fear it may never get done
• Based on discussion I will keep the issue up to date

[xlc]

Some more points:

There could be multiple ways to deliver this:

1. A TS library that allow us to write quick script to invoke it. e.g. Using zx
2. A CLI tool and we use shell script to invoke it
3. A webpage and we have nice GUI

I suggest do 1 first, and we could build 2 or/and 3 based on 1. From UX point of view, 3 will be best as there is one less tool to be installed locally and anyone including non technical people can also use it.

The inputs should be something like:

type Input = {
  networkName: string
  referendumNumber?: number
  proposalHex?: string
  proposalHash?: string
}

And the tool should fetch the proposal from onchain if received a referendum number or the hash.

It should decode the proposal, and determine if it needs to be whitelisted first, and generate a whitelist call if needed.

And it should execute the call on the provide chain, display the storage diff, which will include the events.

It should detect if there is outgoing XCM generated from the proposal, and figure out the networks (using a hardcoded list), and launch a Chopsticks network for the dest networks, and run the XCM, and display the outcome.

It should highlight if a call is scheduled to be dispatched in future, and print out the block number and call hash and bytes.

And the tool should allow user to enter the number and call bytes to dry run the scheduled call.

There could be a mode to allow the tool to automatically do that, but need to handle recurring calls properly.

The output should be displayed in human readable format. Chopsticks already have the ability to decode and generate a nice HTML for viewing.

Some more pointers:

Dry run preimage implementation in Chopsticks https://github.com/AcalaNetwork/chopsticks/blob/master/packages/chopsticks/src/plugins/dry-run/dry-run-preimage.ts

And dru run extrinsic https://github.com/AcalaNetwork/chopsticks/blob/master/packages/chopsticks/src/plugins/dry-run/dry-run-extrinsic.ts

Generate html diff https://github.com/AcalaNetwork/chopsticks/blob/master/packages/chopsticks/src/utils/generate-html-diff.ts

storage diff decoder https://github.com/AcalaNetwork/chopsticks/blob/master/packages/core/src/utils/decoder.ts

There are two ways to interact with a Chopsticks instance:

• Run an instance and use JSON RPC. There are dev RPCs available to dry run or produce blocks. You can use whatever tools to interact with the instance. Useful for say if you want to use Rust to implement this.
• Create a Blockchain instance using setup and interact with the Blockchain instance. This unlocks the full potential of Chopsticks. Also you can do this in a webpage.

[liamaharon]

@PieWol is interested in working on this.

Probably a good place to start would be familiarising with the Chopsticks codebase and this proposal. Ask questions to clear up existing ambiguity about the spec and what we want from a V0.1 of this tool.

Once spec is clear, estimate how long it'll take and let us know an hourly rate you'd charge for this. Finally we can put forward a proposal to the treasury to fund the implementation and also the already done research/discovery work.

Obviously we cannot guarantee treasury will approve funding, but given a clear and reasonable proposal you should expect support from the fellowship to get this funded and implemented.

[flipchan]

@xlc can chopstick verify XCM calls? that would be really useful on the sender and receiver parachains. Given the user knows the predicted output of the xcm call.

[xlc]

xcm is fully supported

[PieWol]

Thank you @liamaharon and @xlc for all the pointers. Let's do the CLI first. I'll need some more time to familiarize myself with the Chopsticks code and to evaluate the effort of the Spec we come up with.

For the CLI it sounds like almost everything is already provided by Chopsticks.

Need to add:

• fetching of proposals
• check for whitelist calls
• event scanning (e.g. scheduled calls and Sent XCM events)
• ability to detect and execute scheduled calls
• a hardcoded list of networks that shall be supported
• automatically boot up a new instance of needed networks
• aggregate storage diff of all the used networks in a human readable format
• enable comparison between expected outcome vs. actual outcome

Given the spec item

Exposes local RPCs allowing devs to connect to and play with post-upgrade state using Polkadot.js

I guess we need to:

Create a Blockchain instance using setup and interact with the Blockchain instance. This unlocks the full potential of Chopsticks. Also you can do this in a webpage.

How about we also add "flagging concerning behaviour" to the Spec? For example xcm calls are given from chain A to chain B and C resulting in B and C sending XCM calls back to Chain A within the same block time. Now the order of execution on chain A will be uncertain.

My apologies if this case is actually already defined and I just didn't know about it yet.

[xlc]

Given the spec item

Exposes local RPCs allowing devs to connect to and play with post-upgrade state using Polkadot.js

I guess we need to:

Create a Blockchain instance using setup and interact with the Blockchain instance. This unlocks the full potential of Chopsticks. Also you can do this in a webpage.

You can but you don't need to.

For example xcm calls are given from chain A to chain B and C resulting in B and C sending XCM calls back to Chain A within the same block time. Now the order of execution on chain A will be uncertain.

How is that relevant to this particular issue?

[PieWol]

How is that relevant to this particular issue?

Misunderstanding. It's not relevant.

[xlc]

Any progress on this? We had too many failed fellowship proposals. If no meaningful amount work is already done, I will just spend a few days to get something out.

[PieWol]

hey @xlc , My progress is here . Feel free to take over and start cooking. No idea what you mean by failed fellowship proposal as I am just doing this in my free time.

[xlc]

This is one failed fellowship proposal that could be avoided with this tool https://collectives.subsquare.io/fellowship/referenda/51

[xlc]

I am building a version at https://github.com/xlc/chopsticks-web

Currently it loads preimages, you can copy & paste it to the dry run panel and run it.

The diff viewer style is a mess but the core functionality works

[xlc]

MVP is done https://xlc.github.io/chopsticks-web/

[h4x3rotab]

MVP is done https://xlc.github.io/chopsticks-web/

This is awesome! Tried on our parachain and it worked like a charm

[liamaharon]

This is awesome, thank you @xlc for this initial effort + detailing a roadmap.

Once the roadmap has been delivered and we add documentation for the tool and how to use it to check opengov proposals, I think we can close this issue.

Ideally the remainder of the roadmap could be delivered by a community member. I will ask around again.