Axe results inconsistencies between "@axe-core/webdriverjs" and "@axe-core/playwright"

soykje commented 1 month ago

Product

playwright

Product Version

^4.9.1

Latest Version

[X] I have tested the issue with the latest version of the product

Issue Description

Expectation

After developping a custom solution for testing a components library using @axe-core/webdriverjs I wanted to improve my project's workflow using Playwright and the dedicated @axe-core/playwright package.

Both were configured with same options:

export const AxeOptions: RunOptions = {
  resultTypes: ['violations', 'incomplete'],
  rules: {
    'page-has-heading-one': { enabled: false },
    'landmark-one-main': { enabled: false },
    region: { enabled: false },
    'color-contrast': { enabled: false },
  },
  runOnly: ['best-practice', 'wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa'],
  reporter: 'v2',
}

As I was using same configuration of the AxeBuilder I expected having same results when testing this page. But for some reason it is not the case :disappointed:

Actual

The custom script actually returns no errors (no violation nor incomplete one). But when I run my Playwright test, I got following results.

How to Reproduce

You can clone the related project. Then:

npm i
npm run test:a11y for Playwright+Axe implementation. The result will be directly logged, and also written as JSON file at public/a11y/a11y-report.json.
npm run a11y:check for custom script execution. If having some selenium-webdriver issues please execute npm run storybook:start then in parallel node bin/check-a11y.js. The result will be directly logged, and also written as JSON file at public/a11y-report.json.

Additional context

As I'm perfectly fine with fixing errors, I'd be very happy to understand the reason of these differences :slightly_smiling_face: Maybe I forgot something in my implementations/instanciations of any of the Axe packages? Anyway, as I'd like to compare both solutions (CI execution time, ...) before eventually going with Playwright one, I'd like to have them working with equivalent configuration!

Thx in advance for all your amazing work, and for your help! :pray:

Zidious commented 1 month ago

👋 Hey @soykje,

Thanks for reporting this. I've noticed a difference in the test setup in the custom script you've linked vs your playwright-test script.

The custom script excludes elements that match '*:not(.docs-story)' as seen in the exclude() call. Your playwright-test analyze() call is not excluding such elements.

I suspect that the reasoning behind the difference in results.

Let me know!

soykje commented 1 month ago

Hello @Zidious :wave:

This difference lies in the fact that in the custom script (let's call it "solution #1" :sweat_smile:) we parse Storybook built pages (such as this one). As you can see, in order to test only Storybook stories, we target these .docs-story blocks. This is why we use .exclude('*:not(.docs-story)'): in order to analyze only the components integrated stories :+1:

In the solution #2 we test the same content but outside of Storybook context. This is why I didn't add the same exclude() call in this last option. Finally, as this last markup wouldn't have any .docs-story, the test would be successful but only because it wouldn't analyze anything :sweat_smile:

Just to be entirely sure of this issue, I ran my Playwright test using VS Code extension. Using the local test environment I get, I tested with different solutions:

With Playwright+Axe test:

I get 2 issues from "incomplete" category:

{
  "@spark-ui/dialog": {
    "timestamp": "2024-08-02T08:30:15.255Z",
    "url": "http://localhost:3002/a11y/dialog",
    "incomplete": [
      {
        "id": "aria-hidden-focus",
        "impact": "serious",
        "tags": [
          "cat.name-role-value",
          "wcag2a",
          "wcag412",
          "TTv5",
          "TT6.a",
          "EN-301-549",
          "EN-9.4.1.2"
        ],
        "description": "Ensures aria-hidden elements are not focusable nor contain focusable elements",
        "help": "ARIA hidden element must not be focusable or contain focusable elements",
        "helpUrl": "https://dequeuniversity.com/rules/axe/4.9/aria-hidden-focus?application=playwright",
        "nodes": [
          {
            "any": [],
            "all": [
              {
                "id": "focusable-modal-open",
                "data": null,
                "relatedNodes": [
                  {
                    "html": "<span data-radix-focus-guard=\"\" tabindex=\"0\" style=\"outline: currentcolor; opacity: 0; position: fixed; pointer-events: none;\" data-aria-hidden=\"true\" aria-hidden=\"true\"></span>",
                    "target": [
                      "span[data-radix-focus-guard=\"\"]:nth-child(1)"
                    ]
                  }
                ],
                "impact": "serious",
                "message": "Check that focusable elements are not tabbable in the current state"
              }
            ],
            "none": [],
            "impact": "serious",
            "html": "<span data-radix-focus-guard=\"\" tabindex=\"0\" style=\"outline: currentcolor; opacity: 0; position: fixed; pointer-events: none;\" data-aria-hidden=\"true\" aria-hidden=\"true\"></span>",
            "target": [
              "span[data-radix-focus-guard=\"\"]:nth-child(1)"
            ]
          },
          {
            "any": [],
            "all": [
              {
                "id": "focusable-modal-open",
                "data": null,
                "relatedNodes": [
                  {
                    "html": "<span data-radix-focus-guard=\"\" tabindex=\"0\" style=\"outline: currentcolor; opacity: 0; position: fixed; pointer-events: none;\" data-aria-hidden=\"true\" aria-hidden=\"true\"></span>",
                    "target": [
                      "span[data-radix-focus-guard=\"\"]:nth-child(6)"
                    ]
                  }
                ],
                "impact": "serious",
                "message": "Check that focusable elements are not tabbable in the current state"
              }
            ],
            "none": [],
            "impact": "serious",
            "html": "<span data-radix-focus-guard=\"\" tabindex=\"0\" style=\"outline: currentcolor; opacity: 0; position: fixed; pointer-events: none;\" data-aria-hidden=\"true\" aria-hidden=\"true\"></span>",
            "target": [
              "span[data-radix-focus-guard=\"\"]:nth-child(6)"
            ]
          }
        ]
      },
      {
        "id": "aria-valid-attr-value",
        "impact": "critical",
        "tags": [
          "cat.aria",
          "wcag2a",
          "wcag412",
          "EN-301-549",
          "EN-9.4.1.2"
        ],
        "description": "Ensures all ARIA attributes have valid values",
        "help": "ARIA attributes must conform to valid values",
        "helpUrl": "https://dequeuniversity.com/rules/axe/4.9/aria-valid-attr-value?application=playwright",
        "nodes": [
          {
            "any": [],
            "all": [
              {
                "id": "aria-valid-attr-value",
                "data": {
                  "messageKey": "controlsWithinPopup",
                  "needsReview": "aria-controls=\"radix-:r3:\""
                },
                "relatedNodes": [],
                "impact": "critical",
                "message": "Unable to determine if aria-controls referenced ID exists on the page while using aria-haspopup: aria-controls=\"radix-:r3:\""
              }
            ],
            "none": [],
            "impact": "critical",
            "html": "<button data-spark-component=\"button\" type=\"button\" class=\"u-shadow-border-transition box-border inline-flex items-center justify-center gap-md whitespace-nowrap px-lg text-body-1 font-bold focus-visible:outline-none focus-visible:u-ring [&amp;:not(:focus-visible)]:ring-inset min-w-sz-44 h-sz-44 rounded-lg bg-main text-on-main hover:bg-main-hovered enabled:active:bg-main-pressed focus-visible:bg-main-focused\" aria-busy=\"false\" aria-live=\"off\" aria-haspopup=\"dialog\" aria-expanded=\"true\" aria-controls=\"radix-:r3:\" data-state=\"open\">",
            "target": [
              "button[aria-haspopup=\"dialog\"]"
            ]
          },
          {
            "any": [],
            "all": [
              {
                "id": "aria-valid-attr-value",
                "data": {
                  "messageKey": "noId",
                  "needsReview": "aria-describedby=\"radix-:r5:\""
                },
                "relatedNodes": [],
                "impact": "critical",
                "message": "ARIA attribute element ID does not exist on the page: aria-describedby=\"radix-:r5:\""
              }
            ],
            "none": [],
            "impact": "critical",
            "html": "<div role=\"dialog\" id=\"radix-:r3:\" aria-describedby=\"radix-:r5:\" aria-labelledby=\"radix-:r4:\" data-state=\"open\" data-spark-component=\"dialog-content\" class=\"z-modal flex flex-col bg-surface group focus-visible:outline-none focus-visible:u-ring max-w-sz-672 fixed top-1/2 left-1/2 -translate-x-1/2 -translate-y-1/2 max-h-[80%] shadow-md rounded-lg data-[state=open]:animate-fade-in data-[state=closed]:animate-fade-out w-full\" tabindex=\"-1\" style=\"pointer-events: auto;\">",
            "target": [
              "#radix-\\:r3\\:"
            ]
          }
        ]
      }
    ],
    "violations": []
  }
}

With custom script:

Finally same as Playwright test! :exploding_head:

{
  "@spark-ui/dialog": {
    "timestamp": "2024-08-02T08:40:39.482Z",
    "url": [
      "http://localhost:3002/a11y/dialog"
    ],
    "incomplete": [
      {
        "id": "aria-hidden-focus",
        "impact": "serious",
        "tags": [
          "cat.name-role-value",
          "wcag2a",
          "wcag412",
          "TTv5",
          "TT6.a",
          "EN-301-549",
          "EN-9.4.1.2"
        ],
        "description": "Ensures aria-hidden elements are not focusable nor contain focusable elements",
        "help": "ARIA hidden element must not be focusable or contain focusable elements",
        "helpUrl": "https://dequeuniversity.com/rules/axe/4.9/aria-hidden-focus?application=webdriverjs",
        "nodes": [
          {
            "any": [],
            "all": [
              {
                "id": "focusable-modal-open",
                "data": null,
                "relatedNodes": [
                  {
                    "html": "<span data-radix-focus-guard=\"\" tabindex=\"0\" data-aria-hidden=\"true\" aria-hidden=\"true\" style=\"outline: none; opacity: 0; position: fixed; pointer-events: none;\"></span>",
                    "target": [
                      "span[data-radix-focus-guard=\"\"]:nth-child(6)"
                    ]
                  }
                ],
                "impact": "serious",
                "message": "Check that focusable elements are not tabbable in the current state"
              }
            ],
            "none": [],
            "impact": "serious",
            "html": "<span data-radix-focus-guard=\"\" tabindex=\"0\" data-aria-hidden=\"true\" aria-hidden=\"true\" style=\"outline: none; opacity: 0; position: fixed; pointer-events: none;\"></span>",
            "target": [
              "span[data-radix-focus-guard=\"\"]:nth-child(1)",
              "span[data-radix-focus-guard=\"\"]:nth-child(6)"
            ]
          }
        ]
      },
      {
        "id": "aria-valid-attr-value",
        "impact": "critical",
        "tags": [
          "cat.aria",
          "wcag2a",
          "wcag412",
          "EN-301-549",
          "EN-9.4.1.2"
        ],
        "description": "Ensures all ARIA attributes have valid values",
        "help": "ARIA attributes must conform to valid values",
        "helpUrl": "https://dequeuniversity.com/rules/axe/4.9/aria-valid-attr-value?application=webdriverjs",
        "nodes": [
          {
            "any": [],
            "all": [
              {
                "id": "aria-valid-attr-value",
                "data": {
                  "messageKey": "noId",
                  "needsReview": "aria-describedby=\"radix-:r5:\""
                },
                "relatedNodes": [],
                "impact": "critical",
                "message": "ARIA attribute element ID does not exist on the page: aria-describedby=\"radix-:r5:\""
              }
            ],
            "none": [],
            "impact": "critical",
            "html": "<div role=\"dialog\" id=\"radix-:r3:\" aria-describedby=\"radix-:r5:\" aria-labelledby=\"radix-:r4:\" data-state=\"open\" data-spark-component=\"dialog-content\" class=\"z-modal flex flex-col bg-surface group focus-visible:outline-none focus-visible:u-ring max-w-sz-672 fixed top-1/2 left-1/2 -translate-x-1/2 -translate-y-1/2 max-h-[80%] shadow-md rounded-lg data-[state=open]:animate-fade-in data-[state=closed]:animate-fade-out w-full\" tabindex=\"-1\" style=\"pointer-events: auto;\">",
            "target": [
              "button[aria-haspopup=\"dialog\"]",
              "#radix-\\:r3\\:"
            ]
          }
        ]
      }
    ],
    "violations": []
  }
}

With Axe extension:

No issues either.

These new errors I can't understand, seem to be directly related to Storybook stories usage. I would have to dig deeper to understand why these errors popped out now, and not before... Anyway would you understand the reason why the Axe browser extension wouldn't give same results? Initially I tried to share same configuration... But maybe I ended with something different?

Anyway, thx in advance for your help :pray:

dequelabs / axe-core-npm