search

component-driven / cypress-axe

Test accessibility with axe-core in Cypress
MIT License
621 stars 86 forks source link

Using runOnly with tags for wcag21 results in passing tests with 0 violations #123

Open tzbarkan opened 2 years ago

tzbarkan commented 2 years ago

If I use runOnly with tags for wcag2a and wcag2aa, I find violations as expected: cy.checkA11y(subject, { runOnly: { type: "tag", values: ["wcag2a", "wcag2aa"] } })

But if I use the tags wcag21a and wcag21aa as documented in axe-core 4.4.1, no violations are found: cy.checkA11y(subject, { runOnly: { type: "tag", values: ["wcag21a", "wcag21aa"] } })

Expectation: using wcag21a and wcag21aa should return a comparable number of violations. Most likely more.

Cypress 9.5.3 Cypress Axe 0.14.0 axe-core 4.4.1 with cypress-watch-and-reload 1.7.4

on Windows 10

AndreaPontrandolfo commented 2 years ago

If I use runOnly with tags for wcag2a and wcag2aa, I find violations as expected: cy.checkA11y(subject, { runOnly: { type: "tag", values: ["wcag2a", "wcag2aa"] } })

But if I use the tags wcag21a and wcag21aa as documented in axe-core 4.4.1, no violations are found: cy.checkA11y(subject, { runOnly: { type: "tag", values: ["wcag21a", "wcag21aa"] } })

Expectation: using wcag21a and wcag21aa should return a comparable number of violations. Most likely more.

Cypress 9.5.3 Cypress Axe 0.14.0 axe-core 4.4.1 with cypress-watch-and-reload 1.7.4

on Windows 10

I'm having a very similar issue. wcag2a level is the only one that works (it gives me a bunch of severity warnings). All the other levels that i tested, just passed without raising warnings.

Cypress 9.7.0 Cypress Axe 0.14.0 axe-core ^4.4.2 Windows 10

bjornlauwerijs commented 2 years ago

I am having the same issue. I expect a higher level to have more issues, but I get none on all tags with wcag21a, wcag21aa, wcag21aaa. I also got some strange results with running wcag2a compared to wcag2aa and wcag2aaa.

Screenshot below explains it better (test is always running on the same page but with different tags).

2022-07-13 10_02_53-IBSBELGIUM_REPORT-screenshot

nurulquamar commented 2 years ago

I am also facing the same issue. wcag21a and wcag21aa are not finding any violations. But when I use wcag2a and wcag2aa I am able to. Also, these violations clearly mention that they are part of wcag21a as well.

axe-core: 4.4.3 cypress-axe: 1.0.0 cypress": 9.5.2 Mac OS 12.6

nurulquamar commented 2 years ago

@tzbarkan I opened an issue with axe-core and it was closed with a comment. See the below link if it helps: https://github.com/dequelabs/axe-core/issues/3760

vladttt1123 commented 5 months ago

can someone tell me what is the benefit of using : values: ['wcag2a', 'wcag2aa'], If as per the documentation that I found:

The WCAG criteria are cumulative—to conform to WCAG Level AA, you must first conform to A. To conform to WCAG Level AAA accessibility criteria, you just first conform to A and AA.

So using only wcag2aa is sufficient, or am I missing something?