Automattic / woocommerce-payments

Accept payments via credit card. Manage transactions within WordPress.
https://wordpress.org/plugins/woocommerce-payments/
Other
176 stars 69 forks source link

Playwright e2e test is flaky: merchant payment gateways confirmation #8875

Open Jinksi opened 6 months ago

Jinksi commented 6 months ago

Description

The Playwright e2e test merchant-payment-gateways-confirmation.spec.ts is failing – inconsistently.

Sometimes it passes on first attempt, sometimes on retry, and sometimes it fails both.

Output from the test failure:

  1) [merchant] › merchant/merchant-payment-gateways-confirmation.spec.ts:79:6 › payment gateways disable confirmation › should disable WCPay after confirming, then enable again without confirming 
    Error: Timed out 10000ms waiting for expect(locator).toHaveText(expected)

    Locator: getByRole('link', { name: '"WooPayments" payment method is currently' })
    Expected string: "Yes"
    Received string: ""
    Call log:
      - expect.toHaveText with timeout 10000ms
      - waiting for getByRole('link', { name: '"WooPayments" payment method is currently' })

      94 |      await waitForToggleLoading( page );
      95 |      await getSaveButton( page ).click();
    > 96 |      await expect( getToggle( page ) ).toHaveText( 'Yes' );
         |                                        ^
      97 |  } );

See playwright-report.zip.

### Tasks
- [ ] https://github.com/Automattic/woocommerce-payments/pull/8899
- [ ] Fix test and re-enable

Additional context

See a list of E2E Playwright GH action runs https://github.com/Automattic/woocommerce-payments/actions/workflows/e2e-pw-pull-request.yml. Failures in this list seem to be caused by this issue alone.

Priority

Low priority, since Playwright e2e tests are non-critical/optional.

Reason why this e2e test is broken

Since this test is disabling/enabling WooPayments as a payment gateway, it may be an uncaught error or timing issue with this process.

On the test site, it's already enabled yet the button appears as disabled.

Running playwright tests locally can show the site is in IDC state.

image

[!Important] Please, ensure when closing this issue (PR fix) that only one e2e: broken label is added and it is accurate.

  • [ ] I confirmed there's only one e2e: broken label in this issue and it is accurate.
Jinksi commented 6 months ago

Re-opening, as we still need to fix this test.