Investigate what tools could be used to automatically check for accessibility issues

Julian commented 8 months ago

Accessibility on the web is, to put it trivially, both incredibly important and also an area that developers (at least this one) know very little about. Specifically, knowing what designs negatively affect accessibility feels really easy to forget about.

Surely there must be a well respected automated tool which we can enable in CI to warn about potential issues. We should find whatever we can easily enable and have it run as part of the UI tests.

I (Julian) am very much not a frontend expert, but I know the ARIA guidelines are one text reference that's relevant. Perhaps there are tools which enforce some of its recommendations.

At very least some simple checks like "ensure elements always have alt text" seem like they must exist.

sudo-jarvis commented 8 months ago

Hi @Julian

I've done some research and found a few widely used tools that support accessibility testing:

Tools for Accessibility Testing

Google Lighthouse: GitHub Link
- Lighthouse is an open-source, automated tool for improving the quality of web pages. It audits web pages for performance, accessibility, SEO, and more.
Pa11y: GitHub Link
- Pa11y is an open-source automated accessibility testing tool. It runs accessibility tests on pages via the command line or Node.js, so one can automate the testing process.
axe-core: GitHub Link
- Axe is an accessibility testing engine for websites and other HTML-based user interfaces. It's fast, easy to use, and works with all modern web technologies.

Recommendation

Out of these, Lighthouse seems to be the best accessibility testing tool. It's developed by Google and widely used. Additionally, it offers other features like performance testing and SEO optimization testing. It supports testing for both desktop and mobile.

Also, there are multiple related tools that work on top of lighthouse and enhance its functionalities such as : auto-crawler to crawl all pages present on a website, automated lighthouse-ci-action, etc. Reference: Related Projects

How Lighthouse Works

Lighthouse uses Puppeteer in the background. It opens the specified URL in an automated browser, tests for the various issues and then creates a report highlighting those issues.

You can test how lighthouse works by going on to bowtie's website and then opening chrome dev tools, there is already a separate lighthouse tab present where the report can be generated.

Testing

You can also test lighthouse by using its cli tool on https://bowtie.report by running:

npm -g install lighthouse

lighthouse https://bowtie.report --only-categories=accessibility --output=json --chrome-flags="--headless" --form-factor=desktop --screenEmulation.disabled

Sample Response

"button-name": {
      "id": "button-name",
      "title": "Buttons do not have an accessible name",
      "description": "When a button doesn't have an accessible name, screen readers announce it as \"button\", making it unusable for users who rely on screen readers. [Learn how to make buttons more accessible](https://dequeuniversity.com/rules/axe/4.8/button-name).",
      "score": 0,
      "scoreDisplayMode": "binary",
      "details": {
        "type": "table",
        "headings": [
          {
            "key": "node",
            "valueType": "node",
            "subItemsHeading": {
              "key": "relatedNode",
              "valueType": "node"
            },
            "label": "Failing Elements"
          }
        ],
        "items": [
          {
            "node": {
              "type": "node",
              "lhId": "1-0-BUTTON",
              "path": "1,HTML,1,BODY,0,DIV,0,DIV,0,NAV,0,DIV,4,BUTTON",
              "selector": "div > nav.navbar > div.container-fluid > button#theme-toggler",
              "boundingRect": {
                "top": 25,
                "bottom": 61,
                "left": 533,
                "right": 577,
                "width": 44,
                "height": 36
              },
              "snippet": "<button id=\"theme-toggler\" class=\"btn border-0 me-1\">",
              "nodeLabel": "div > nav.navbar > div.container-fluid > button#theme-toggler",
              "explanation": "Fix any of the following:\n  Element does not have inner text that is visible to screen readers\n  aria-label attribute does not exist or is empty\n  aria-labelledby attribute does not exist, references elements that do not exist or references elements that are empty\n  Element has no title attribute\n  Element's default semantics were not overridden with role=\"none\" or role=\"presentation\""
            }
          }
        ],
        "debugData": {
          "type": "debugdata",
          "impact": "critical",
          "tags": [
            "cat.name-role-value",
            "wcag2a",
            "wcag412",
            "section508",
            "section508.22.a",
            "TTv5",
            "TT6.a",
            "EN-301-549",
            "EN-9.4.1.2",
            "ACT"
          ]
        }
      }
    }

Challenges

From the first look, it looks like these tools work on the finally built html files and thus require the webpage url or html file to work. Maybe would need to run the frontend locally during the CI to be able to test them in this manner.
Identifying where the issue actually occurs in our react code could be a problem as can be seen from the sample output.

Julian commented 8 months ago

Thanks for the first research -- sounds a bit like https://github.com/bowtie-json-schema/bowtie/issues/963#issuecomment-1968960969 -- I should hope there's a general way to plug tools in as part of the vite/react build process.

Lighthouse seems like it does more than just accessibility as you say, but however we figure out the first bit (run a tool as part of building the site) I'm sure it will be easy to try out the actual output from all 3 and seeing how useful they look.

bowtie-json-schema / bowtie