Flaky E2E: User sees welcome banner after signup

worldomonation commented 2 years ago

Spec file

wp-invite__new-user.ts

Suite or step

User sees welcome banner after signup

Framework

Playwright

Logs

Logs:

page.waitForSelector: Timeout 30000ms exceeded.
=========================== logs ===========================
waiting for selector ":has-text("You're now an Editor of: ")" to be visible
  selector resolved to 11 elements. Proceeding with the first one.
============================================================
at Object.<anonymous> (/home/teamcity-2/buildAgent/work/c4a9d5b38c1dacde/test/e2e/specs/specs-playwright/wp-invite__new-user.ts:103:15)
    at Promise.then.completed (/home/teamcity-2/buildAgent/work/c4a9d5b38c1dacde/node_modules/jest-circus/build/utils.js:390:28)
    at callAsyncCircusFn (/home/teamcity-2/buildAgent/work/c4a9d5b38c1dacde/node_modules/jest-circus/build/utils.js:315:10)
    at _callCircusTest (/home/teamcity-2/buildAgent/work/c4a9d5b38c1dacde/node_modules/jest-circus/build/run.js:218:40)
    at _runTest (/home/teamcity-2/buildAgent/work/c4a9d5b38c1dacde/node_modules/jest-circus/build/run.js:155:3)
    at _runTestsForDescribeBlock (/home/teamcity-2/buildAgent/work/c4a9d5b38c1dacde/node_modules/jest-circus/build/run.js:66:9)
    at _runTestsForDescribeBlock (/home/teamcity-2/buildAgent/work/c4a9d5b38c1dacde/node_modules/jest-circus/build/run.js:60:9)
    at _runTestsForDescribeBlock (/home/teamcity-2/buildAgent/work/c4a9d5b38c1dacde/node_modules/jest-circus/build/run.js:60:9)
    at run (/home/teamcity-2/buildAgent/work/c4a9d5b38c1dacde/node_modules/jest-circus/build/run.js:25:3)
    at runAndTransformResultsToJestFormat (/home/teamcity-2/buildAgent/work/c4a9d5b38c1dacde/node_modules/jest-circus/build/legacy-code-todo-rewrite/jestAdapterInit.js:167:21)
    at jestAdapter (/home/teamcity-2/buildAgent/work/c4a9d5b38c1dacde/node_modules/jest-circus/build/legacy-code-todo-rewrite/jestAdapter.js:82:19)
    at runTestInternal (/home/teamcity-2/buildAgent/work/c4a9d5b38c1dacde/node_modules/jest-runner/build/runTest.js:389:16)
    at runTest (/home/teamcity-2/buildAgent/work/c4a9d5b38c1dacde/node_modules/jest-runner/build/runTest.js:475:34)
    at Object.worker (/home/teamcity-2/buildAgent/work/c4a9d5b38c1dacde/node_modules/jest-runner/build/testWorker.js:133:12)

TeamCity ID: 6965438

worldomonation commented 2 years ago

Just to be clear, the user is failing to see the welcome banner after signup.

worldomonation commented 2 years ago

Has occurred 6/287 runs, but 3/6 were due to different reasons.

2/3 were due to a similar but slightly different message: The requested invite was not found.

dpasque commented 2 years ago

Wowsers, this one is tricky...

So, in a couple of these cases, there's something screwy with getting redirected to a second login form. That's a separate issue to look into.

However, with this case of the link being bad, here's what I've been able to figure out...

The error is that the invite link has already been used. The response from the web request to accept the invite link:

{
    "error": "invalid_input_invite_used",
    "message": "The invite has already been used. Please ask the site owner to send another."
  }

There is no other concurrent tests that could account for this failure
When we get the link from the mailosaur inbox, we are using newly created email address that is unique
None of these invited users are listed on the invited blog as having accepted or even as being pending
I tried querying the DB with the key for the invitation, but in all cases, no invite was found with that key... Which is weird, because the backend code should then be returning a different error 😬

dpasque commented 2 years ago

Next steps for this issue:

Watch for another test to fail with this
Grab the invite key and query to see it's status in the DB

If that continues to not be found, there's probably some kind of bug in creating and fetching invites that needs to be debugged, and it might be best to pull in some people who know more about that area.

worldomonation commented 2 years ago

Moving back into To Do since not much we can do here for now.

worldomonation commented 2 years ago

We've worked around this issue by forcibly dismissing the popup.

See https://github.com/Automattic/wp-calypso/blob/57e3319cf3e8318c8329819bc1dff6e23ae33349/packages/calypso-e2e/src/lib/components/editor-welcome-tour-component.ts for details.

Automattic / wp-calypso