Is your feature request related to a problem? Please describe.
Due to variability, Lighthouse CI may sometimes report a lower performance score that causes an assertion to fail, whereas for most runs, the assertion passes. Since the Lighthouse test takes rather long, I've configured numberOfRuns: 1 for my CI pipeline. However, if, and ideally only if, a performance assertion fails, it would be helpful if the test could be re-run for a given URL.
Increasing numberOfRuns isn't quite the same thing, as it will run multiple times also when the first test passes.
Describe the solution you'd like
An option to retry the assertions for a given page if the the first run fails.
Describe alternatives you've considered
Wrapping the entire LHCI run to re-run if it fails (this needlessly tests URLs again that have already passed).
Is your feature request related to a problem? Please describe.
Due to variability, Lighthouse CI may sometimes report a lower performance score that causes an assertion to fail, whereas for most runs, the assertion passes. Since the Lighthouse test takes rather long, I've configured
numberOfRuns: 1for my CI pipeline. However, if, and ideally only if, a performance assertion fails, it would be helpful if the test could be re-run for a given URL.Increasing
numberOfRunsisn't quite the same thing, as it will run multiple times also when the first test passes.Describe the solution you'd like
An option to retry the assertions for a given page if the the first run fails.
Describe alternatives you've considered
Wrapping the entire LHCI run to re-run if it fails (this needlessly tests URLs again that have already passed).