Shareable performance testing utils

[swissspidy]

What problem does this address?

The core performance team wants to pave the way for more projects to adopt performance testing as part of their development workflow. So far we've added Playwright support to @wordpress/scripts and contributed the very useful Metrics and Lighthouse fixtures to @wordpress/e2e-test-utils-playwright. The latest addition to the fixtures is #55660. While that works great so far, this is only part of the toolchain.

Gutenberg, WordPress core, and third-party project all use a similar toolchain for running browser-based performance tests:

• Playwright (via @wordpress/scripts or directly)
• Dedicated utils to get metrics (from @wordpress/e2e-test-utils-playwright)
• Some glue code to save metrics as test attachments
• A custom test reporter to get results from attachments and save them to a file
• A custom CLI script to further parse and & format that results file and optionally compare it with results from a previous run/version
  • Gutenberg even has a script that automatically runs two tests for the comparison, other projects like core do this via GitHub Actions right now

To help make things easier, I've started creating an example repository of what is involved, as well as an easy-to-use GitHub Action for performance testing. Especially when looking at the latter you can see there is a lot of custom code, a lot of repetition.

So, how can we address this?

What is your proposed solution?

We could build a standardized performance test reporter and CLI script(s) and make that available through @wordpress/e2e-test-utils-playwright or perhaps a new @wordpress/performance-tests package. This way, developers who want to set up performance tests don't have to copy all of that boilerplate, and benefit from updates as well.

An alternative would be to have the community come up with such a package. Core and Gutenberg could keep their existing scripts as they are rather unique projects who probably want to retain more customization.

Maybe there is also no need at all and folks are happy with just a GitHub Action as an abstraction.

[swissspidy]

cc @kevin940726 @WunderBart @Mamaduka

[felixarntz]

I don't feel strongly about whether we use @wordpress/e2e-test-utils-playwright or a new @wordpress/performance-tests (that will likely depend on the first in some way). The latter may make more sense to clarify the purpose, as the "e2e test" term doesn't indicate that it's about performance tests/benchmarks at all. But the decision also depends on what makes sense technically.

I do think we should provide utilities beyond just packaged up in a GitHub Action. A reusable GitHub Action like the one you've created definitely should be part of the overall solutions offered, but that could probably rely on those scripts coming from a separate package. The main reason I think we also need to provide the scripts themselves in a reusable way is that there are more custom use-cases where maybe a one-size-fits-all action isn't feasible, but a one-size-fits-all set of scripts might. For example for WordPress core, which doesn't use @wordpress/env, but still uses Playwright and would probably benefit from a unified set of performance benchmarking scripts/tests.

[WunderBart]

Pinging @youknowriad, as the performance.js script was originally planned to be a standalone package (@wordpress/performance-tests?) for running perf tests against any project. I think it's still worth pursuing, so there's likely some overlap there.

[kevin940726]

I don't have strong preference here. Welcome any contribution! I think it's okay to place it in e2e-test-utils-playwright as long as it's optional and has to be opted-in. We might want to rename the package to something like playwright-utils anyway in the future, but that's undecided.