search

GoogleChrome / lighthouse

Automated auditing, performance metrics, and best practices for the web.
https://developer.chrome.com/docs/lighthouse/overview/
Apache License 2.0
28.27k stars 9.35k forks source link

Lighthouse fails to produce accessibility score altogether for incomplete axe-core result #15828

Open rajsite opened 7 months ago

rajsite commented 7 months ago

FAQ

URL

https://axecore-lighthouse-failure.glitch.me/

What happened?

Running lighthouse on the above page fails to produce any accessibility score at all:

image

What did you expect?

It would be preferable for lighthouse to be robust to errors in axe-core and still produce an accessibility score.

We try to gate code submissions on accessibility scores and while lighthouse / axe-core reports plenty of other successful accessibility checks, it is completely masked by the incomplete check (in-terms of gating on accessibility checks and tracking history of the score).

I would expect:

What have you tried?

Isolated and disabled lighthouse testing for pages using the component that causes the lighthouse failure. Reported the incomplete result and reproducing example to axe-core: https://github.com/dequelabs/axe-core/issues/4335 But would still like lighthouse to be more robust to axe-core errors and generate a useful score (might be a bit lower because of the error or maybe the error does not contribute to the score).

How were you running Lighthouse?

PageSpeed Insights, Chrome DevTools

Lighthouse Version

11.3.0

Chrome Version

121.0.6167.185

Node Version

No response

OS

No response

Relevant log output

axe-core Error: Cannot read properties of undefined (reading 'props')

as mentioned above, the axe-core error was reported at: https://github.com/dequelabs/axe-core/issues/4335 but this issue is for improving lighthouses score calculation robustness to axe-core errors

adamraine commented 7 months ago

Thanks for filing a bug with axe-core, we will close this once it's fixed and the dependency is updated.

An accessibility score to still be calculated to prevent breaking PR gates completely based on score and fail to see general progress

If an axe check throws an error, we don't know if the audit passes and so we can't know what the accessibility score would be.

Maybe we can treat the audit as failing which could be less punishing than having no score at all, but I'm not completely sold on the idea.

A link / more resources to report the incomplete result to axe-core

I think this could be useful. 90% of the time issues related to accessibility we wind up upstreaming to axe-core.

rajsite commented 7 months ago

Maybe we can treat the audit as failing which could be less punishing than having no score at all, but I'm not completely sold on the idea.

In our case we had to disable the CI gate for the accessibility check for several weeks (until we had time to come up with a reproducing example and make the reports) in which time other accessibility issues that would have been caught otherwise were allowed through.

adamraine commented 7 months ago

We are not going to change how the accessibility category is scored since there is value in displaying errors loudly. Reporting this error to us when it happens is exactly what we want :)

If you weren't doing this already, you can use skipAudits to disable a single erroneous audit without disabling the entire accessibility score check.

I will leave this issue open because there is still an upstream issue and we should update the accessibility category description and docs to mention Axe and direct users to file bugs there.

paulirish commented 7 months ago

@rajsite It was fun to walk back the history in https://github.com/ni/nimble/issues/1650 and see everything. In this case I think you managed to find a really nice bug in axe. It's wildly infrequent we see those sorts of DOM level exceptions from axe. So.. hats off to you for really stressing the system. :)