search

treosh / lighthouse-ci-action

Audit URLs using Lighthouse and test performance with Lighthouse CI.
MIT License
1.17k stars 82 forks source link

Wild variances in v3 #65

Closed Snugug closed 3 years ago

Snugug commented 4 years ago

I'm using v3, running a live url 3 times. The URLs are all hitting Firebase's hosting CDN, so they're very stable performance-wise. In my latest run, my Performance score across 3 runs was 24, 73, 73. The exact same site, with identical code, run from the same infra, but at a different URL is reporting values of 89, 74, and 72. When running Lighthouse from Chrome, the same URLs consistently have a performance score of between 85-87. I'm curious as to why there is so much variance in the actions version of Lighthouse as compared to the browser version, and if there's any way to stabilize the results.

Thanks!

exterkamp commented 4 years ago

Do you have the benchmark indices of those variant runs? Or know the time the github-action took to run?

I know in the past sometimes github had bad days and actions VMs were really flaky. #14

alekseykulikov commented 4 years ago

Yes, would be interesting to see benchmarkIndex values. They should be around 750 to be trustable. We need to enable PSI as an alternative runner https://github.com/GoogleChrome/lighthouse-ci/issues/251

alekseykulikov commented 3 years ago

Let's keep the track of variance in #14