Integrate Cypress E2E browser testing

Problem

Converged components are becoming more complex with respect to interactions. The interactions that Menu and Accordion support are equally important to their design variations. Unit testing interactions can only go so far and have the following problems:

JS DOM is simply a mock and is not the real browser
As test components get more complicated, it gets harder to debug unit tests
In a unit test every DOM element is always accessible if it is rendered (even if is not visible or covered by another element)
Unit test examples are always simpler than full examples the developer uses to manually test behaviour

Fluent UI also supports visual regression tests through Screener. Screener is great a what is does and a very solid way of testing components with lots of design variations like the current converged components that already exist: Button, Avatar, Link. However these problems occur when trying to test interactions with screener:

Cannot be reproduced locally
Expensive to maintain because of the above point
Assertions must be screenshots - doesn't make sense if multiple sets of interaction paths have the same result
Cannot detect focused element - must rely on visual cues

Appetite

Future interaction-heavy components will be introduced in Fluent UI convergence such as Popup, Input, Dialog. It should be easy to write examples for these test and run the same kind of manual validations that the developer will inevitably make to prevent regressions to interactions.

Solution

Integrate https://www.cypress.io/ in the repo. Tests should be written against the storybook instances that are currently used during component development. The official storybook documentation even suggests using cypress as one of the solutions for interaction testing

The ideal requirements are described below

Running tests locally

Tests should be run locally with a simple yarn <command> command

Scoping/filtering tests

Tests can be scoped and filtered in the same way that it is possible to run unit tests for an entire test suite or a single test in a suite

Running tests in CI

E2E tests should be run on CI and follow the scoping currently applied with Lage so that only affected projects tested. The possibility of running E2E tests on the deployed PR storybooks should be considered.

The extra time required in CI should be measured, this project proposes comparing a set of tests that assert the same things using both screener and cypress, using the Menu and Accordion components, to see using what final cost this will be for our CI.

Minimize maintenance effort

Native behaviour should be used as much as possible so that it is easily upgradeable without any custom layers.

Pros/Cons

Cons

Cypress only supports chromium based browsers such as edge/chrome/electron
Cypress uses its own test framework and mocha - inconsistent with the current jest setup that we have
Will add extra time to our CI
Cypress does not remove VR testing - it will become additional work for some components
Tests should not depend on component code+utilities since cypress bundles everything into the browser

Pros

Browser tests that can be reproduced and debugged locally
Can assert rendered DOM and focus without relying only on screenshots
Leverages the existing storybook workflow that is currently used during development
Will be easier to scope during CI for affected packages than current screener tests
Cypress bundles everything into the browser, there is no 'magic' layer that controls the browser.

Discarded solutions

Puppeteer is also another well known browser E2E framework that is used with the chrome browser. v0 previously relied on this solution and discarded it in favour of cypress due better debugging features that cypress has such as triggering the debugger or pausing tests to allow manual interaction.

Similar to the above Playwright is a first party solution similar to puppeteer but provides multi-browser tests not only for chrome but also Firefox and Safari. Playwright is not opinionated on the test runner and can be integrated with a jest testing flow. This solution was also discarded mainly due to the more advanced debugging capabilities that cypress offers. However maintaining actual multi browser support is not a trivial effort so it is hard to say if this advantage of playwright would actually be used.

Chromatic also offers a visual testing solution for storybook. However this solution is rather for VR testing similar to Screener. Chromatic does not support interactions and advises consumers not to use them in VR testing. The storybook official documentation even suggests using cypress for interaction testing.

Risks

This project could potentially add more unnecessary devops work and test maintenance to our plate for the future. Hopefully this risk can be reduced as much as possible by building as little custom layers over cypress as possible and relying only on native (easily upgradeable) behaviour and testing on existing platforms (react-examples storybook).

Blurring the lines between types of testing. It can be argued that if cypress is simple to use then it is no longer so clear why unit test tests are needed, if interactions can be tested more reliably in the browser. This is a fair point and should be addressed as we leverage E2E testing more.

Likewise, clear boundaries to be set with Screener. VR testing is essential for a design system, VR coverage should not suffer just because there is another way to test components in the browser. The proposed project is intended to complement VR testing.

Out of scope (No-gos)

Replacing Screener with Cypress snapshots
Converting unit tests to E2E tests
Writing full sets of E2E tests for all existing components

microsoft / fluentui