Closed MikeMcC399 closed 1 year ago
@larswmh / @dsarkar
I don't have a solution for this problem, as when I tested it, it was working on GitHub and I don't exactly know the reason for the failure.
https://github.com/corona-warn-app/cwa-website/actions/runs/4859232685/jobs/8661692312 from the blog update worked.
So the test is just flaky, not failing always.
Would you like to re-open this issue?
The test seems to hang up waiting for /en/faq/results/faq_duplicate.json
to load and process. I could not see any obvious way so far to solve this issue. 🙁
To fix this issue it probably needs src/assets/js/app.js debugging and fixing.
It may be simplest to just disable the test cypress/e2e/faq_link_attr.cy.js from running in CI workflows. It does usually work when it is run manually though.
@MikeMcC399 Ok, thanks for the extensive analysis. I think your suggestion to disable the flaky test in CI makes sense.
@dsarkar
Thank you keeping me in the loop. I decided not to spend more time on this after disabling the test in CI, since the test is still available for local manual testing and I would not expect any new FAQ articles to be published. I did however take a brief look at the new PR and the changes make it more difficult to see where the error came from if there is an error. It's up to you if you want to implement these new changes. I can't really say if the analysis of the root cause is correct.
Hi @MikeMcC399 I have also tested it and with the log output I have the impression that we will be better able to track down the error if something else should occur.
Hi @mtb77
Thanks for looking at this issue. It seems that I was misled by the error message Timed out after waiting
100000msfor your remote page to load.
If softExpect
is changed to expect
I found that it works. softExpect
is actually an unsupported extension to Cypress which was implemented through a "monkey patch" in cypress/support/e2e.js. Using it means that each result is stored and the test continues even if there is an error, so that at the end it can report multiple errors. It does mean that it consumes memory and the latest changes increased the number of links to check by 40% which was apparently enough to kill it! 🙁
At this stage of the game I suggest to go with whatever works best for you. I originally wrote the test, so I have just learned that it was not scalable. Sorry for that!
Thanks for the feedback @MikeMcC399 - and I learned something again! The timeouts were also really mysterious. I would then suggest merging and then we at least know that we can open another feature request at Cypress to include softExpect in the core without "monkey patch" 😎
Cypress test cypress/e2e/faq_link_attr.cy.js is sometimes failing when running on GitHub using the workflows:
The test is attempting to visit
baseUrl
:http://localhost:8000
+/
+lang
:en
+faqResultsUrl
:/faq/results
=
http://localhost:8000/en/faq/results
and failing.
Manual reproduction
The first time this is run, the duration is 01:27 (min:sec).
Now run again:
This time it only takes 00:06 (min:sec).
The issue can occur locally with browser-sync or in production.
Suggestion
Similar issues have been seen with the website previously and the root cause of the issues was not resolved (see for instance https://github.com/corona-warn-app/cwa-website/issues/3071 and https://github.com/corona-warn-app/cwa-website/issues/3070). This is not an error in the test itself, however a workaround can be to visit the root of the website
/
before each test. This seems to reset whatever it is that is causing the web not to respond in the complex FAQ section.Internal Tracking ID: EXPOSUREAPP-15135