Make e2e tests more sustainable

[agryaznov]

Currently ink-waterfall tests have the following drawbacks (regarding the CI job running them):

1. A test could be easily broken by some change on the UI side, which is out of our control, while it would still block all ink! PRs because of non-green CI.

Currently e2e tests are simply being run versus currently deployed version of polkadot-js. The good effect of this is that it lets us catch any issue which ink! (and pallet_contract) users face while interacting with them on https://polkadot.js.org/apps, which is by now the most likely usage case. However, once these tests run on the CI, this some(not so rare)times leads to a PR stuck for ink!, which, again, could be caused by a bug\change on a UI side. So basically we are maintaining tests for the UI part, at the costs of longer PR delivery for our products.

Possible actions to solve this could be:

• Make UI projects run contracts e2e tests (on ink! examples) as part of CI\CD process of their deployment. Solely depends on @jacogr for polkadot-js, and on @statictype for contracts-ui.
• As part of the CI job run on a ink! PR, run ink-waterfall tests versus a concrete UI version, specially deployed for that particular run as part of that CI pipeline.
• Make ink-waterfall use cargo-contract for interactions with contracts. We could make this option to be used in ink! PR CI jobs.

2. Quite unsustainable test results like this one are noticed, which generally dissapear after one or a couple of restarts.

thread 'tests::erc721::erc721_operator_approvals' panicked at 'upload must have succeeded, but events contain only []', src/uis/polkadot_js.rs:470:9

or

 thread 'tests::erc1155::erc1155_approvals' panicked at 'upload must have succeeded, but events contain only []', src/uis/polkadot_js.rs:470:9=

These ones could suddenly happen on different examples on different runs, and just a restart is an only so far known way to treat this. I think we should look into it and make it more sustainable.

[jacogr]

I ripped out all e2e tests on my side around 18 months (2 years?) ago since it adds no benefit to my side and it was something dropped in my lap with a “promise to maintain” which don’t happen.

There is only one of me with -

• the api
• the util libs
• the apps UI
• the extension
• the cli tools
• the wasm libs
• … I probably forgot some

The above applies to all: coding, reviews, documentation, support, etc.

The “UI” component is one small portion of repos where I’m the only developer, and contracts are not touched at all.

With a team, yeah sure, but not going to add more to my pile of shit to get done.

(And I honestly do appreciate issues that have been logged from tests here in the past - they saved me a lot of extra unneeded issues and even fuller queues to look at - the last one picked up from here would have certainly resulted in all hell breaking loose all-over)

[jacogr]

If you are running these on all PRs, make it a nightly job that logs an issue when it fails. CI PR pipeline stays fast. (Proper failures are easily tracked to merges in the last 24)

[statictype]

E2E tests for ink! examples in Contracts UI are underway. I think ink-waterfall should move away from UI testing and only test with cargo-contract which now has all the needed features