Investigate move to more modern tools for our in-browser tests

[romaricpascal]

What

Puppeteer has its limitations when it comes to running our in-browser tests. We'd be looking to move away from it towards a more modern tool like Cypress or Playwright.

~Pick 1 or 2 components and try rewriting their behavioural tests using the two frameworks.~

Re-implement tests in both tool for:

• the CharacterCount JS component, which has client side interactions
• the Checkboxes Nunjucks tempate, as a way to weigh the benefits of having template tests as browsers tests to help debugging

Demo back to the rest of the team.

Why

We want to understand if using Cypress or Playwright (or similar) can make it easier for us to test in multiple browsers, giving us confidence to ship the upcoming changes to the way we write JavaScript.

We also think this can help us with managing session state between tests.

Assumptions

Timebox

We should review progress after this period of time has elapsed, even if the spike has not been 'completed'

2 days

Who is working on this?

Spike lead: Colin

Spike buddy: Romaric

Questions to answer

• [ ] In which browsers do each tool let us run our tests? Are there limitations?
• [ ] How long does it take to run tests in each tools, compared to the current tool?
• [ ] What's the reporting experience locally?
• [ ] What's the reporting experience on CI?
• [ ] What's the debugging experience locally?
• [ ] What's the debugging experience on CI?
• [ ] What's the authoring experience?
• [ ] Can we get code coverage from the tests? If yes, can we merge it with coverage from the other tests?
• [ ] Bonus: Is there any help for automating accessibility tests?
• [ ] Based on these criteria, which of the two tools (if any) would be best suited for us?

Done when

You may find it helpful to refer to our expected outcomes of spikes.

• [ ] Cypress tests for CharacterCount JS are implemented
• [ ] Cypress tests for Checkboxes macro are implented
• [ ] Playwright tests for CharacterCount JS are implemented
• [ ] Playwright tests for Checkboxes macro are implented
• [ ] Stretch WebDriver tests for CharacterCount JS are implemented
• [ ] Stretch WebDriver tests for Checkboxes macro are implemented
• [ ] Questions have been answered or we have a clearer idea of how to get to our goal
• [ ] Findings have been reviewed and agreed with at least one other person
• [ ] Findings have been shared, e.g: via a write-up on the ticket, at a show & tell or team meeting

[romaricpascal]

As a tangential thought, we could look into how our tests are split to ensure that we rely on heavy things like browser tests and review apps only when necessary. As a strawperson split, for discussion:

1. End to end browser tests relying on user-level interactions ("Click on button with this text", "Type in field with this label") and verifications ("The screen displays X", "The browser announced Y" if possible...) rather than lower level use of IDs and classes
2. Technical check of IDs attributes and classes without necessarily spinning up a whole browser
3. Unit tests to check response of individual functions/methods to ranges of input

Which tool we pick shouldn't affect such reorganisation should we chose to go for it.

This may tie more into https://github.com/alphagov/govuk-frontend/issues/2565

[romaricpascal]

An extra tool we may want to investigate in a separate spike would be https://webdriver.io/. Authoring and debugging experience look a bit more dev-y (no GUI to replay the tests at first glance), but the use of the WebDriver protocol would open the door to a wider range of browsers using Browserstack Automate or Saucelabs

[romaricpascal]

Sharing some notes and links to code:

• [x] https://github.com/alphagov/govuk-frontend/pull/3064
• [x] https://github.com/alphagov/govuk-frontend/pull/3162
• [ ] [Dropped due to limited browser support] The Cypress implementation

[romaricpascal]

Investigation is done, newer work will be tracked by #3278.