Ability to automatically run several "rounds" of the Lighthouse test in DevTools and get an averaged result

[josephrocca]

Feature request summary Ability to automatically run several "rounds" of the Lighthouse test and get an averaged result (plus access to data on the individual rounds so you can see how much variation there is).

The variability guide already recommends that users do this - but it would be great to add this as an option in DevTools.

What is the motivation or use case for changing this? Lighthouse test results can vary quite a bit, even when running them one after the other. I think this is especially the case for pages that have ads, because sometimes different types of ads (e.g. animated/video vs static/image) will be loaded which affects the performance score (not just the Publisher Ads plugin score).

How is this beneficial to Lighthouse? The goal of Lighthouse is to help developers test performance improvements that they've made to their sites/apps. In a recent test I ran Lighthouse 4 times, and got these scores: 60, 62, 58, 56. With such large variation in the scores, it's possible that a new performance improvement that I added to my app isn't captured in the score after one test - the large variation could end up convincing me that I've actually decreased performance. It's only after averaging over a few tests that it's clear that performance was improved. Manually running several tests and aggregating the results is time-consuming compared to what could be a simple button click.

[connorjclark]

Previous time this was discussed: #10825

All we did last time was encode our variance advice into a function we expose in our node package (#10859)

I'm still open to a feature in DevTools to run LH a number of times and show the average.

[connorjclark]

We aren't all reaching consensus on this exact feature (automating running multiple times), but we want to explore something else that may fill some of the same use case:

• replace the LHR report select dropdown with a much better UI / history view, giving a summary of the reports so you can better understand the multiple runs/the variance of your page wrt. performance.
• maybe also change from session-based storage for these reports to persisting to disk

[josephrocca]

Can I ask what the argument against this is? As a dev I just want to click a button and get a score that I can trust. Putting the burden on devs to click and wait and click and wait, etc. until they've got a statistically significant value seems like unnecessarily bad UX. I'm sure you've discussed this, but I'm curious what is causing the lack of consensus. Thanks!