TenKeyLabs / dappwright

🏌🏼‍E2E testing for dApps using Playwright + MetaMask & Coinbase Wallet
https://www.npmjs.com/package/@tenkeylabs/dappwright
Other
82 stars 21 forks source link

Metamask version issue #344

Closed alejogranerorequest closed 2 months ago

alejogranerorequest commented 2 months ago

I have a lot of instabilities with the last version 2.8.4, the recommended version is 11.16.3 and the last one available in Chrome is 11.15.6

@osis , could you downgrade the recommended version to 11.15.6?

alejogranerorequest commented 2 months ago

It can't finish the wallet setup, it gets stuck after importing the wallet using the secret words; I can't switch networks or accounts.

alejogranerorequest commented 2 months ago

test-failed-4 It fails after this page, I took it from the trace log.

alejogranerorequest commented 2 months ago

I still face instabilities on CI, do you know if something is missing @osis ?

johanneskares commented 2 months ago

Had the same issue, I created a lib that uses a headless mock wallet for testing: https://github.com/johanneskares/wallet-mock

osis commented 2 months ago

Hmmm seems like there is some new variable behaviour in the setup. Should be able to replicate in the tests for the project.

osis commented 2 months ago

I wouldn't necessarily classify as a version issue, more of a random case that isn't handled. But you can always downgrade the version your running in the config.

osis commented 2 months ago

I still face instabilities on CI, do you know if something is missing @osis ?

there are known issues with playwright and popups in github actions. I suggest setting up your CI tests to be similar to this repo (retry logic) until they are solved. Looks like running the local test suite is stable.

osis commented 2 months ago

test-failed-4 It fails after this page, I took it from the trace log.

When it comes to this screen in particular though. I did find it odd since it's not a popup so I looked a bit deeper. Looks like this is a bit of a bug(?) in MetaMask where the behaviour changes only for the first wallet setup after the extension has been installed.

I could replicate locally by manually removing the extension and running the tests locally. The reason why I suspect it's a bug is because after bumping to 11.16.13 it went back to being stable. Will publish a release but you can also specify the newer version of MetaMask in your config/launch options.

Wagalidoom commented 2 months ago

I still face instabilities on CI, do you know if something is missing @osis ?

there are known issues with playwright and popups in github actions. I suggest setting up your CI tests to be similar to this repo (retry logic) until they are solved. Looks like running the local test suite is stable.

Hi @osis ! Could you provide more information about this please? I set up 3 retries in the playwright.config.ts but still facing issues on Github Actions (it works properly on local). Thank you

osis commented 2 months ago

@Wagalidoom you can take a look at https://github.com/TenKeyLabs/dappwright/blob/5f9d1e0c01cbbeeccfc681340aaadf937d929d8d/.github/workflows/test.yaml#L28-L59

osis commented 2 months ago

going to mark this as closed as I can't replicate this specific issue anymore.