Add visual regression testing with Playwright

Eric-Arellano commented 1 year ago

See https://github.com/pradyunsg/furo/discussions/661.

Visual regression testing takes screenshots of portions of the site. With every change, we will know if that screenshot has changed or not. If the change was intentional, we update the screenshot; else we have to fix the regression.

Example failure:

 1) snapshots.test.js:3:5 › footer snapshot has not changed ───────────────────────────────────────

    Error: Screenshot comparison failed:

      809 pixels (ratio 0.01 of all image pixels) are different.

...

    Expected: /app/snapshot_results/snapshots-footer-snapshot-has-not-changed/footer-snapshot-has-not-changed-1-expected.png
    Received: /app/snapshot_results/snapshots-footer-snapshot-has-not-changed/footer-snapshot-has-not-changed-1-actual.png
        Diff: /app/snapshot_results/snapshots-footer-snapshot-has-not-changed/footer-snapshot-has-not-changed-1-diff.png

      4 |   await page.goto("");
      5 |   const footer = page.locator("footer");
    > 6 |   await expect(footer).toHaveScreenshot();
        |                        ^
      7 | });
      8 |

...

Uses Docker for consistency

Websites render slightly differently between operating systems, e.g. Linux and macOS. Naively, this would cause visual regression tests to fail when running the tests locally on a Mac or Windows.

So, we use Docker to make sure we always use the same environment when rendering the docs. (We still build the docs locally.)

Users who don't want to run Docker locally can rely on CI to get the updated snapshots via GitHub Actions Artifacts.

More advanced users can also run the tests locally. They need to install Docker and have the Docker daemon running. But otherwise, our scripts automate everything. They don't need to know how to use Docker. They only need to run npm test.

How to run locally

The user has to first build the docs with nox -e docs, which builds the docs locally. Then, they only run npm test.

Playwright will start up a server for us.

If the user changes the theme, they must rebuild the docs with nox -e docs. nox -e docs-live does not work because it does not write its result to build/docs.

We could simplify it so that users only need to run npm test, without first running nox -e docs. This would make the workflow slower because we'd need to install Nox and copy over all the project files into Docke, then run Sphinx inside Docker, which is slower on my M1. But it's a tradeoff, and that may be better.

Why Playwright?

A popular alternative for visual regression testing is using Jest's Snapshot testing with https://github.com/americanexpress/jest-image-snapshot and Puppeteer. But I had major issues getting Puppeteer to build on my M1 in Docker, since they don't release Linux ARM64 wheels for Puppeteer.

Beyond the M1 issues, I found Playwright is a more ergonomic test runner than Jest for visual regression tests.

Eric-Arellano commented 5 months ago

Closing due to it being stale and not having time to refresh this. But hopefully it can help others interested in adding regression testing. We have found it incredibly useful at qiskit-sphinx-theme.

pradyunsg commented 5 months ago

I'll probably build off of this when I make some time for this theme -- thanks @Eric-Arellano for filing this and apologies for not being able to get back to you sooner!

pradyunsg / furo