Investigate odd placement of cypress failures reports

[swirtSJW]

When investigating cypress failures on tugboat, it has become challenging to find the output about the failures because it seems to be getting interwoven with other tests running. Hunch: related to parallelization?

Example 1

Example 2

Example 3

Tasks:

• [ ] Determine the cause of output confusion

Team

Please check the team(s) that will do this work.

• [x] CMS Team
• [ ] Public Websites
• [ ] Facilities
• [ ] User support

[jilladams]

Potentially related thread: Public Websites had a Tugboat continually failing to build a FE with Cypress test errors: https://dsva.slack.com/archives/CT4GZBM8F/p1697838105032549

[BerniXiongA6]

Hi @ndouglas -- could you help triage this one to get it into prerefinement activities? Thank you!

[ndouglas]

@BerniXiongA6 This is a legit issue, and it can be aggravating. It's due to an upstream issue in cypress-parallel, I believe. @ariperez is the best situated to look at it, but he might appreciate a break from dealing with Cypress or might not be comfortable making a PR on an upstream Node project. I can look at it too, but it's pretty far down on my list of priorities at this point. Another possibility is to completely rework how we run Cypress, e.g. reëvaluate Sorry-Cypress or consider adopting Cypress's commercial offering for orchestrating Cypress parallelization, which obv should be turnkey and would probably be resolved more quickly but would cost money.

[ndouglas]

Hey team! Please add your planning poker estimate with Zenhub @ariperez @edmund-dunn @JunTaoLuo @olivereri @teeshe

[olivereri]

I've been noodling on this when I saw this issue raised in August and the subsequent fix. Seems not as satisfactory to still be missing richer browser features.

Question: If we had Sorry-Cypress up and running but kept the current Cypress-parallel setup the same but modified the api_url in /root/.cache/Cypress/10.8.0/Cypress/resources/app/packages/server/config/app.yml would Cypress-parallel report the sorry-cypress dashboard? It probably needs the specific Sorry-Cypress cypress-cloud npm package to send everything needed.

I'm assuming that any solution with Sorry-Cypress is going to require a custom cypress parallelization solution? Cypress-Parallel is able to spread test across multiple threads on a single CI machine right?

[ndouglas]

I haven't dug deeply into how cypress-parallel does what it does, but I believe it uses a different mechanism than the "Director" system in Cypress and Sorry-Cypress. I believe it just splits up the specs according to the weighting and sends each to a different thread, rather than Cypress acting like a client and asking the Director for a new job to perform. I could be off, of course.

I'm assuming that any solution with Sorry-Cypress is going to require a custom cypress parallelization solution? Cypress-Parallel is able to spread test across multiple threads on a single CI machine right?

Yep, exactly.