search

dequelabs / axe-core

Accessibility engine for automated Web UI testing
https://www.deque.com/axe/
Mozilla Public License 2.0
5.95k stars 774 forks source link

Update frame-tested rule to always run #3123

Open straker opened 3 years ago

straker commented 3 years ago

The frame-tested rule is a rule that ensures all iframes on the page have been tested. If an iframe is unable to run axe.run for any reason, the rule will return as needs review and list the iframe that was not tested for accessibility issues.

The frame-tested rule is currently listed as a best-practice rule. This means there are configurations that allow axe to run without reporting errors from iframes. For example: axe.run({ runOnly: 'labels' }) will only run the label rule and not frame-tested. The problem this causes is that if there is an iframe on the page with an invalid label and axe.run fails to run in that iframe, the user would never know. We would report 0 violations when in reality there is one (so a false negative).

We should update the frame-tested rule so that it always runs and cannot be disabled since it is an error reporting rule (it either passes or returns as needs review). To do this we should remove the best-practice tag (leaving the category and review-item tags) and ensure axe always run review-item tags regardless of the user config.

WilcoFiers commented 2 years ago

I'm reluctant to make this change as proposed. There might be scripts out there trying to run a single rule, if those all of a sudden get results back from two rules rather than one, we might introduce some bad breaking changes.

I think a better solve here is that either we give this one the wcag2a tag, which everyone who wants to run axe pretty much always runs, or we give it all the wcag2 tags, as well as best practice so that no matter what level runs, this rule is on, but then if you try to run a single rule, or if you try to run rules for a single success criterion the rule is off.

dbjorge commented 3 months ago

I'm not so sure "always run" would be an anti-feature like Wilco is describing. I definitely believe it would be for some users, but I remember back when I worked on Accessibility Insights adding specific logic wrapping axe-core so that anytime we did try to run just a single rule, it also ran frame-tested.