joemcgill
commented
3 months ago Thanks for kicking this discussion off, @swissspidy.
I appreciate the distinction between two main objectives. For now, I'm going to limit my thoughts to the first objective, "Improve Core/GB performance testing".
One of the things that I observed during the 6.5 release is that finding the source of a server timing regression was challenging when that regression was committed to the Core repo as part of a larger Gutenberg sync. I think we can improve this somewhat by updating the performance tests in the Gutenberg repo to include the same server timing metrics that we record for each commit to the WP Core repo (i.e., wpTotal, wpBeforeTemplate, and wpTemplate). While TTFB is a close proximity, the additional noise from network requests and calculating the metric in a headless browser makes pinpointing potential regressions more difficult.
I also strongly agree with your suggestion to improve our demo content, and think this applies as much to the Core tests as the Gutenberg tests. Currently in Core, we are only testing the default homepage for the Twenty Twenty-One and Twenty Twenty-Three themes after importing a Theme Test Data from this commit. While this keeps test content consistent over time, there are a number of limitations to this approach including the fact that some specific use cases that we care about (e.g., measuring the effect of image optimizations on LCP) are not covered by our current test content.
From a visualization point of view, our current dashboards at codevitals.run have become harder to use over time as we've added additional metrics. I'd love to investigate improving or replacing these dashboards with a system where we could more granularly filter results by metric, theme, template, object cache, etc. In doing so, we should also evaluate how we are normalizing and storing the raw data which currently gets normalized before it's saved to the dashboard's database meaning we don't have the ability to build new reports using the original unfiltered data.
Last idea for now, is that when we introduced these tests we used Twenty Twenty-One and Twenty Twenty-Three as representative themes for classic and block themes, but that has proven to be overly simplistic as some performance regressions are only visible based on characteristics of a theme (e.g., how many template part variations they register). At minimum, we should add Twenty Twenty-Four to our test matrix in both repos.
TL;DR:
- Add server timing metrics to the GB performance workflow
- Improve demo content for the performance workflow in both repos
- Add specific use cases (e.g., homepage, page with large LCP image, etc) to our workflows for both repos
- Improve the dashboard for tracking performance over time so it's more user friendly
- Improve data collection process to make raw data queryable in the future
- Add tests for Twenty Twenty-Four
swissspidy
Last year we made several big improvements in the performance testing area. To name a few:
849
852
As a reminder, the two main objectives we had:
While progress slowed down due to other priorities, there is still a lot of untapped potential this year. I am therefore opening this issue to kickstart a discussion on next steps.
Some enhancements/features that are already on our radar:
Server-Timing, etc.Some loose ideas:
Curious to hear everyone's thoughts :)