search

dequelabs / axe-core-npm

Mozilla Public License 2.0
595 stars 68 forks source link

@axe-core/cli@4.4.2 reports 0 violations when adding tags #508

Open stefansurkamp opened 2 years ago

stefansurkamp commented 2 years ago

Product: cli

Expectation: axe should show the same (or more) errors when settings --tags wcag2a.

Actual: axe reports 0 violations found! when adding --tags anything.

Motivation: Adding a certain set of tags should still allow for proper execution.

axe-core version: 4.4.1
@axe-core/cli: 4.4.2

- Node version: v14.18.2
- Platform:  Mac

Reproduction:

▶ axe google.com
Running axe-core 4.4.1 in chrome-headless
Testing http://google.com ... please wait, this may take a minute.
[...]
9 Accessibility issues detected.
Testing complete of 1 pages

▶ axe google.com --tags wcag2a
Running axe-core 4.4.1 in chrome-headless
Testing http://google.com ... please wait, this may take a minute.
 0 violations found!
Testing complete of 1 pages

▶ axe google.com --tags cheers
Running axe-core 4.4.1 in chrome-headless
Testing http://google.com ... please wait, this may take a minute.
 0 violations found!
Testing complete of 1 pages
Zidious commented 2 years ago

Hey @stefansurkamp,

Thanks for submitting this issue.

We will investigate this issue and add validation to tags if required.

stefansurkamp commented 2 years ago

Thanks for the quick response. I did some investigation myself and want to report that this might be a non-issue - it might only be "0 violations" for google.com specifically. I tried out another URL

▶ axe https://diluz.io/sebastian
Running axe-core 4.4.1 in chrome-headless

Testing https://diluz.io/sebastian ... please wait, this may take a minute.
[...]
15 Accessibility issues detected.
Testing complete of 1 pages

▶ axe https://diluz.io/sebastian --tags wcag2a
Running axe-core 4.4.1 in chrome-headless

Testing https://diluz.io/sebastian ... please wait, this may take a minute.
[...]
14 Accessibility issues detected.
Testing complete of 1 pages

So it seems, everything's working as expected.

Am I then right in the assumption that not providing tags, will apply all of them, while having tags will only apply a subset?

Zidious commented 2 years ago

Correct, not applying the --tags flag will apply all of the axe-core tags, minus experimental rules. Applying a subset of these tags, the CLI will run all the rules associated with those tags. Example:

All WCAG2A rules:

axe www.deque.com --tags wcag2a

All WCAG2A, WCAG2AA, WCAG2AAA rules:

axe www.deque.com --tags wcag2a,wcag2aa,wcag2aaa

However, providing an invalid tag (example: --tags random) does not default back to all tags nor currently, do any type of tag validation. This is something we may need to investigate.