Command runner action

[xlc]

Setup a GH action to execute command

The action should read a config.yml from this repo, which contains various configs including whitelisted gh user and issue numbers.

Format

The gh action should be triggered by new comment on either PR or Issue. The comment body should follow the following format:

/bot [CMD] [ARGS...]
ENV1=VAL1
ENV2=VAL2

The comment should be started with /bot or it should be ignored. Once a command is accepted, The bot should immediately make a replay with the command to be run, and run the command, and report the status by modifying the replied issue. In case the format is invalid, it should reply with an error message and a help message. Each CMD can have zero or more space separated arguments. Except the first line, the rest of the body follows the .env format and should be loaded to the command. This env should override the block numbers specified via the update-env script.

Each task have a permission (who can call it), a trigger (issue or pr), an output (sent via reply), an action(e.g. merge PR)

Commands

Run test

• Command: run
• Argument: a name to filter which test to run
• Trigger: Whitelisted issues specified in config.yml. This allows people to watch this repo but unsubscribe from the test run issue to avoid spam
• Permission: Anyone
• Output: The test result
• Action: None

This allows people to trigger a test run. Useful to validate everything still works after a runtime upgrade, or run a test with a new wasm.

Update test

• Command: update
• Argument: a name to filter which test to run
• Trigger: Whitelisted issues specified in config.yml
• Permission: Anyone
• Output: The test result and snapshot diff
• Action: Create a new PR from the snapshot diff

This allows people to update test snapshots in case it changed. For example, after a runtime upgrade, the fees, weights, event names etc could change. People should review the diff in the PR to ensure all the changes are expected.

Merge PR

• Command: merge / cancel-merge
• Argument: None
• Trigger: PR
• Permission: Whitelisted gh users specified in config.yml
• Output: None
• Action: Enable / disable auto-merge https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/incorporating-changes-from-a-pull-request/automatically-merging-a-pull-request#enabling-auto-merge

This allows external teams to trigger merge PR by put their gh username in config.yml without permission management on the gh repo.

Bump KNOWN_GOOD_BLOCK_NUMBERS.env

Depends on https://github.com/open-web3-stack/e2e-test/issues/4

• Command: bump
• Argument: None
• Trigger: Whitelisted issue
• Permission: Anyone
• Output: The test result
• Action: Update, commit and push the change

This allows people to manually bump the KNOWN_GOOD_BLOCK_NUMBERS.env. The action should reject the command that specifies additional env as we should only test and bump the good block numbers using the default env.

References:

Acala Benchmark action: https://github.com/AcalaNetwork/Acala/blob/master/.github/workflows/benchmark.yml Polkadot Fellowship RFC action: https://github.com/polkadot-fellows/RFCs/blob/main/.github/workflows/rfc-action.yml

[bkchr]

• Whitelisted issues specified in config.yml

Can you give an example on what issues would be whitelisted? I don't understand what you mean by this.

Bump KNOWN_GOOD_BLOCK_NUMBERS.env

Let's assume a runtime upgrade happened. How do I execute this? Do I create an issue and then trigger the bot there?

[xlc]

Can you give an example on what issues would be whitelisted? I don't understand what you mean by this.

We can just create an issue Bot trigger or something like that and put the issue number in the config and people can trigger the bot in that issue. This particular issue will be spammy but people can unsubsribe from this issue to not get notified from it.

Let's assume a runtime upgrade happened. How do I execute this? Do I create an issue and then trigger the bot there?

On the Bot trigger issue, anyone can just make a comment /bot bump and the bot will be triggered to run the tests.

[acatangiu]

Do you expect to have the hw capacity to support the whole ecosystem to constantly run these tests in this repo’s CI using the bot?

I mean that would be great! But also very expensive? Could be funded from treasury. Or this could be pushed downstream:

• this repo CI runs once a day (or some other low frequency periodic interval) as continuous regression testing
• On-demand tests are done downstream by teams/users cloning the repo and running “locally”

Instead of the bot, this would need framework(/scripts) to easily run locally.

[xlc]

it is just a yarn test to run the tests locally

the free github action runner should be able to run the tests, just it could take long time. it shouldn’t be a big issue if people use the filter feature to only run a subset of tests.

we can also provide a more powerful runner (same setup as we do for ORML) and only allow waitlisted users to trigger it to avoid unexpected high server cost