[REQUEST] Research and comment on accessibility testing tools for potential integration with Theme Builder

[PaulaPaul]

Suggestion/Concern

The goal of this issue is to research how accessibility testing can be:

• integrated into the design and implementation of new websites and design systems that leverage Theme Builder
• integrated into the evaluation of existing web sites and design systems, and how those existing design systems can be improved using Theme Builder

The outcome of this research should include:

• An initial evaluation of source tooling such as
  • Purple A11y
  • Axe Core
  • Chromatic
• Notes for a potential blog post on the topic of accessibility testing when using Theme Builder
• Thoughts on next steps and tasks that can be created to integrate these tools with Theme Builder end to end design and testing

Proposed Solution

Evaluation observations, suggestions and notes for blogs and potential next steps should be included as comments on this issue, for review by the Theme Builder core maintainers.

[aaronreed708]

@lwnoble has a contact at Chromatic. Currently they do isolated, component-level testing, but not testing in context. We can ask them if they are interested in expanding the scope of their product (with our assistance). Otherwise probably not the best tool for us to use.

[kburk1997]

Commented on the Design Ops issue, but also would comment here:

Testing:

• Accessibility Insights for Web - free Chrome/Edge extension that contains automated, guided, and manual test abilities. Very comprehensive - lots of tests that are available for free and it has a GitHub integration too that makes it easy to file issues. There is also tooling available for Windows + Android apps.
• axe-devtools - freemium offering - Chrome extension with only automated tests are free. axe API is free up to a point - its Developer Hub offering is paid. Also has mobile offerings.
• @storybook/addon-a11y - Storybook accessibility plugin - runs automated checks using axe api

Linting:

• axe-linter - VSCode extension free, GitHub actions integration paid
• eslint-plugin-jsx-a11y - plugin for ESLint - allows us to catch common accessibility issues + extend the configuration to map Material UI elements to HTML elements

For Designers:

• axe for Designers - contains accessibility annotations + auto scan functionality
• Stark - color contrast tool for designers - includes a lot more than just color contrast, including functionality for developers
• Web Accessibility Annotation Kit - accessibility annotations toolkit for Figma from CVS Health

TODO find plugins for Penpot

Integration Testing:

• Cypress - offers an OSS Plan for open source projects for its premium offering, but basic functionality is also free
  • Beta Cypress Accessibility offering that has a waitlist
  • Accessibility testing can be achieved using plugins such as:
  • [axe Developer Hub]() - axe offering that can integrate an accessibility scan into all Cypress tests
  • cypress-axe - barebones integration with axe api
  • cypress-audit - not up to date but does let you run Pa11y or Lighthouse
• Playwright - free open source offering
  • Playwright guide for automated accessibility testing
• Selenium WebDriver - multiple language bindings for browser automation
  • Accessibility testing can also be done using 3rd party plugins like axe-selenium-java
  • Very barebones compared to Cypress + Playwright

Unit testing - jest-axe