Flaky Cypress test faq_link_attr.cy.js

[MikeMcC399]

Cypress test cypress/e2e/faq_link_attr.cy.js is sometimes failing when running on GitHub using the workflows:

• .github/workflows/build-and-test.yml or
• .github/workflows/cypress-test-prod.yml

  1) Check attributes of FAQ links
       Check links are using _blank to open in new frame
         Test links for _blank presence:
     CypressError: Timed out after waiting `100000ms` for your remote page to load.

Your page did not fire its `load` event within `100000ms`.

You can try increasing the `pageLoadTimeout` value in `cypress.config.js` to wait longer.

Browsers will not fire the `load` event until all stylesheets and scripts are done downloading.

When this `load` event occurs, Cypress will continue running commands.
      at timedOutWaitingForPageLoad (http://localhost:8000/__cypress/runner/cypress_runner.js:137295:72)
      at <unknown> (http://localhost:8000/__cypress/runner/cypress_runner.js:138269:16)
      at tryCatcher (http://localhost:8000/__cypress/runner/cypress_runner.js:8914:23)
      at <unknown> (http://localhost:8000/__cypress/runner/cypress_runner.js:4036:41)
      at tryCatcher (http://localhost:8000/__cypress/runner/cypress_runner.js:8914:23)
      at Promise._settlePromiseFromHandler (http://localhost:8000/__cypress/runner/cypress_runner.js:6849:31)
      at Promise._settlePromise (http://localhost:8000/__cypress/runner/cypress_runner.js:6906:18)
      at Promise._settlePromise0 (http://localhost:8000/__cypress/runner/cypress_runner.js:6951:10)
      at Promise._settlePromises (http://localhost:8000/__cypress/runner/cypress_runner.js:7027:18)
      at _drainQueueStep (http://localhost:8000/__cypress/runner/cypress_runner.js:3621:12)
      at _drainQueue (http://localhost:8000/__cypress/runner/cypress_runner.js:3614:9)
      at ../../node_modules/bluebird/js/release/async.js.Async._drainQueues (http://localhost:8000/__cypress/runner/cypress_runner.js:3630:5)
      at Async.drainQueues (http://localhost:8000/__cypress/runner/cypress_runner.js:3500:14)
  From Your Spec Code:
      at eval (webpack:///./cypress/e2e/faq_link_attr.cy.js:20:19)
  at Array.forEach (<anonymous>)
      at eval (webpack:///./cypress/e2e/faq_link_attr.cy.js:18:22)
      at itCallback (webpack:///./cypress/support/e2e.js:74:4)
      at Context.eval (webpack:///./cypress/support/e2e.js:95:31)

The test is attempting to visit

• baseUrl: http://localhost:8000 +
• / +
• lang: en +
• faqResultsUrl: /faq/results

= http://localhost:8000/en/faq/results

and failing.

Manual reproduction

npm ci
npm run build
npx start-server-and-test start-server http://localhost:8000 "cypress run -s 'cypress/e2e/faq_link_attr.cy.js' --e2e --headless --browser chrome"

The first time this is run, the duration is 01:27 (min:sec).

Now run again:

npx start-server-and-test start-server http://localhost:8000 "cypress run -s 'cypress/e2e/faq_link_attr.cy.js' --e2e --headless --browser chrome"

This time it only takes 00:06 (min:sec).

The issue can occur locally with browser-sync or in production.

Suggestion

Similar issues have been seen with the website previously and the root cause of the issues was not resolved (see for instance https://github.com/corona-warn-app/cwa-website/issues/3071 and https://github.com/corona-warn-app/cwa-website/issues/3070). This is not an error in the test itself, however a workaround can be to visit the root of the website / before each test. This seems to reset whatever it is that is causing the web not to respond in the complex FAQ section.

---

Internal Tracking ID: EXPOSUREAPP-15135

[MikeMcC399]

@larswmh / @dsarkar

• Unfortunately, the workaround https://github.com/corona-warn-app/cwa-website/pull/3480 has failed 🙁 See https://github.com/corona-warn-app/cwa-website/actions/runs/4858683797.

I don't have a solution for this problem, as when I tested it, it was working on GitHub and I don't exactly know the reason for the failure.

[MikeMcC399]

https://github.com/corona-warn-app/cwa-website/actions/runs/4859232685/jobs/8661692312 from the blog update worked.

So the test is just flaky, not failing always.

Would you like to re-open this issue?

[MikeMcC399]

The test seems to hang up waiting for /en/faq/results/faq_duplicate.json to load and process. I could not see any obvious way so far to solve this issue. 🙁

[MikeMcC399]

To fix this issue it probably needs src/assets/js/app.js debugging and fixing.

It may be simplest to just disable the test cypress/e2e/faq_link_attr.cy.js from running in CI workflows. It does usually work when it is run manually though.

[MikeMcC399]

• PR https://github.com/corona-warn-app/cwa-website/pull/3487 is a workaround, which disables the test.

[dsarkar]

@MikeMcC399 Ok, thanks for the extensive analysis. I think your suggestion to disable the flaky test in CI makes sense.

[dsarkar]

FYI PR https://github.com/corona-warn-app/cwa-website/pull/3490

[MikeMcC399]

@dsarkar

Thank you keeping me in the loop. I decided not to spend more time on this after disabling the test in CI, since the test is still available for local manual testing and I would not expect any new FAQ articles to be published. I did however take a brief look at the new PR and the changes make it more difficult to see where the error came from if there is an error. It's up to you if you want to implement these new changes. I can't really say if the analysis of the root cause is correct.

[mtb77]

Hi @MikeMcC399 I have also tested it and with the log output I have the impression that we will be better able to track down the error if something else should occur.

[MikeMcC399]

Hi @mtb77

Thanks for looking at this issue. It seems that I was misled by the error message Timed out after waiting100000msfor your remote page to load.

If softExpect is changed to expect I found that it works. softExpect is actually an unsupported extension to Cypress which was implemented through a "monkey patch" in cypress/support/e2e.js. Using it means that each result is stored and the test continues even if there is an error, so that at the end it can report multiple errors. It does mean that it consumes memory and the latest changes increased the number of links to check by 40% which was apparently enough to kill it! 🙁

At this stage of the game I suggest to go with whatever works best for you. I originally wrote the test, so I have just learned that it was not scalable. Sorry for that!

[mtb77]

Thanks for the feedback @MikeMcC399 - and I learned something again! The timeouts were also really mysterious. I would then suggest merging and then we at least know that we can open another feature request at Cypress to include softExpect in the core without "monkey patch" 😎

[MikeMcC399]

• Since https://github.com/corona-warn-app/cwa-website/pull/3490 stops this issue occurring, it can be closed.