search

alphagov / accessibility-tool-audit

Automated accessibility tools audit
https://alphagov.github.io/accessibility-tool-audit/
MIT License
80 stars 30 forks source link

Seems like aXe catches many more problems than listed #15

Closed tinynow closed 6 years ago

tinynow commented 7 years ago

Using the aXe browser plugin, many more of errors are detected that you did not list. I did not run an automated test, but it uses the same core...

Examples: html lang element, empty title...

screenshot of google chrome 8-2-17 12-33-20 pm

selfthinker commented 6 years ago

This is most probably due to aXe having got better since we last tested it (which was over a year ago). We plan to retest some tools and add new ones at some point. Pull Requests are also always welcome.

selfthinker commented 6 years ago

I have just re-tested everything in aXe and the results are a bit different but not a lot. See #17 for more details. On the one hand aXe and most other tools generally find more issues than we have on the page, partly because some of them are duplicated. On the other hand I don't get the same number of violations as in your screenshot. How did you get that number? What did you test? Did you test with the Chrome extension or any other version of aXe? (I get 36 violations and 38 "needs review" when I test the big long page, which is not all of the issues because some are on separate pages.)

tinynow commented 6 years ago

I only used the Chrome extension. It's been a while, but I will try to replicate my results.

dervism commented 6 years ago

When using the axe-cli on the command line, I get "55 accessibility issues detected". I am running the following command:

npm install axe-cli -g

then

axe https://alphagov.github.io/accessibility-tool-audit/test-cases.html --timeout 60000 --tags wcag2a,wcag2aa,section508,best-practice,experimental

With all categories (gives the same result):

axe https://alphagov.github.io/accessibility-tool-audit/test-cases.html --timeout 60000 --tags wcag2a,wcag2aa,section508,best-practice,experimental,cat.aria,cat.color,cat.text-alternatives,cat.time-and-media,cat.tables,cat.semantics,cat.sensory-and-visual-cues,cat.parsing,cat.structure,cat.name-role-value,cat.keyboard,cat.forms,cat.language

'55' in this context is the sum of all the occurrences of every violation: https://github.com/dequelabs/axe-cli/blob/develop/index.js#L88

The actual number of unique violations using axe-core 2.5.0 is 39. Using the Chrome extension (axe-core version 2.4.0), I get as @selfthinker says only 36.

marcysutton commented 6 years ago

The aXe extension doesn't run experimental rules or all of the best practices, so turning those tags on in axe-cli will cause a pretty large difference.

selfthinker commented 6 years ago

Interesting! What is the recommended way and all the recommended settings for aXe to catch the most issues? Is it what @dervism posted?

marcysutton commented 6 years ago

Yep! You can enable tags in axe-core itself, or through axe-cli or axe-webdriverjs. If you're looking for a browser extension with more results, aXe Coconut is our prerelease extension with Shadow DOM support and experimental rules enabled.

selfthinker commented 6 years ago

Thanks, I might re-run everything with the experimental settings on then.

selfthinker commented 6 years ago

I have just re-run all the non-positive tests with aXe Coconut and only found 4 more issues. So, it's definitely an improvement but not such a big difference as the above indicates. I will update the relevant results soon.

A general remark on how we test and why that might not have much to do with any summarised number you get out of a tool... We always test every single page (i.e. 142 pages) and not the big page where everything is shown together. And then we check if an issue is found or not, and if it is, we check if it is the issue we were expecting. For example, the focus not visible test contains a link with a target of #. Although aXe rightly says that that would be a problem, it's not the problem we were looking for.

Unfortunately, the test pages generally lack a lot of context because it is just much easier to write them that way. But that also means that tools pick up on such things.

So, if aXe says it found 60 issues, it will be some of the issues we were looking for and other issues we inadvertently created but which are not part of the issues we checked.

selfthinker commented 6 years ago

I have just updated the aXe results (see #27). In the end there it found 3 more issues (not 4 as I wrote earlier). But that pushed it to a (joint) 3rd place!

I will close this issue now. I think the difference in the numbers comes from misunderstanding how we count the issues.