Closed Eric-Arellano closed 5 months ago
Closing due to it being stale and not having time to refresh this. But hopefully it can help others interested in adding regression testing. We have found it incredibly useful at qiskit-sphinx-theme.
I'll probably build off of this when I make some time for this theme -- thanks @Eric-Arellano for filing this and apologies for not being able to get back to you sooner!
See https://github.com/pradyunsg/furo/discussions/661.
Visual regression testing takes screenshots of portions of the site. With every change, we will know if that screenshot has changed or not. If the change was intentional, we update the screenshot; else we have to fix the regression.
Example failure:
Uses Docker for consistency
Websites render slightly differently between operating systems, e.g. Linux and macOS. Naively, this would cause visual regression tests to fail when running the tests locally on a Mac or Windows.
So, we use Docker to make sure we always use the same environment when rendering the docs. (We still build the docs locally.)
Users who don't want to run Docker locally can rely on CI to get the updated snapshots via GitHub Actions Artifacts.
More advanced users can also run the tests locally. They need to install Docker and have the Docker daemon running. But otherwise, our scripts automate everything. They don't need to know how to use Docker. They only need to run
npm test
.How to run locally
The user has to first build the docs with
nox -e docs
, which builds the docs locally. Then, they only runnpm test
.Playwright will start up a server for us.
If the user changes the theme, they must rebuild the docs with
nox -e docs
.nox -e docs-live
does not work because it does not write its result tobuild/docs
.We could simplify it so that users only need to run
npm test
, without first runningnox -e docs
. This would make the workflow slower because we'd need to install Nox and copy over all the project files into Docke, then run Sphinx inside Docker, which is slower on my M1. But it's a tradeoff, and that may be better.Why Playwright?
A popular alternative for visual regression testing is using Jest's Snapshot testing with https://github.com/americanexpress/jest-image-snapshot and Puppeteer. But I had major issues getting Puppeteer to build on my M1 in Docker, since they don't release Linux ARM64 wheels for Puppeteer.
Beyond the M1 issues, I found Playwright is a more ergonomic test runner than Jest for visual regression tests.