Component accessibility testing

[xman343]

Changes

Adding a Cypress script that tests all the components in the examples folder for accessibility violations using cypress-axe.

Testing

To run all tests headlessly: yarn test --filter=a11y from root.

To run interactively inside browser (locally): yarn workspace a11y open.

Docs

N/A

[mayank99]

I thought about it some more and here's what I'd suggest:

• Rename the playwright-testing workspace to a11y and move it into a new folder called testing. After that, update the root package.json workspaces to use testing/*. The idea behind this is that we'll be able to add more types of testing in the future (likely using playwright too).

• Instead of manually listing out all components and examples, lets change the approach to loop over all examples. So you will have a single test file that looks something like:

  import * as React from 'react';
  import { test, expect } from '@playwright/experimental-ct-react';
  import AxeBuilder from '@axe-core/playwright';
  import * as allExamples from 'examples';
  
  test.describe('Should have no WCAG violations', () => {
  Object.entries(allExamples).forEach(([name, Component]) => {
    test(name, async ({ mount, page }) => {
      await mount(<Component />);
      const results = await new AxeBuilder({ page }).analyze();
      expect(results.violations).toHaveLength(0);
    });
  });
  });

  The benefit of this is less manual work, both upfront and ongoing. Any future examples will be automatically tested without needing to add new tests.

• Use a github action workflow to automatically runs these tests in CI on every push. You can add new job in build.yml.

[mayank99]

I looked into this some more and it appears that "component testing" in playwright is very limited. Playwright needs to know all the components beforehand, so my above suggestion about looping will not work. Related issue: https://github.com/microsoft/playwright/issues/18057

So your options are:

• either list out everything manually, or
• get rid of all "component testing" configuration, and instead use regular "end to end testing". This will require manually wiring up all the boilerplate needed to mount the components, but it should be doable.

I would suggest going with the latter. It can be achieved in three steps:

1. Create a simple isolated "test app", similar to the playwright/index.html that currently exists. You can use either Vite or Astro for this.
2. Set up the test app such that it renders all the examples at different urls. For example, localhost:6666/AlertMainExample, localhost:6666/AlertInlineExample, etc.
   • If using astro, dynamic routes might come in handy.
   • You'll have to research and experiment yourself. It doesn't matter how you do this - all that matters is that there is a different url for each example.
3. Within playwright, loop over all of those examples again, but instead of mounting the component, use page.goto to visit all of those urls and run Axe against them.

[mayank99]

Another option you might want to explore is Cypress component testing. It is considered stable, unlike Playwright component testing (which is still experimental), so it's possible that you might be able to loop over components in cypress just fine. Definitely worth experimenting. If you run into similar limitations, then you can carry on with my suggestion above about using playwright e2e testing instead.

Note: You'll have to use cypress-axe instead of @axe-core/playwright.

[mayank99]

Here's the code I used to get cypress ct working:

cypress.config.js

```js import { defineConfig } from 'cypress'; import react from '@vitejs/plugin-react'; import { createRequire } from 'node:module'; const require = createRequire(import.meta.url); const axeCorePath = require.resolve('axe-core'); export default defineConfig({ component: { devServer: { framework: 'react', bundler: 'vite', viteConfig: { plugins: [react()], }, }, env: { axeCorePath }, video: false, screenshotOnRunFailure: false, setupNodeEvents(on, config) { on('task', { log(message) { console.log(message); return null; }, table(message) { console.table(message); return null; }, }); }, }, }); ```

cypress/support/component.ts

```ts import { mount } from 'cypress/react18'; declare global { namespace Cypress { interface Chainable { mount: typeof mount; } } } Cypress.Commands.add('mount', mount); import '@itwin/itwinui-react/styles.css'; import 'cypress-axe'; ```

cypress/support/component-index.html

```html

a11y testing

```

all.cy.tsx

```tsx import * as React from 'react'; import * as allExamples from 'examples'; import { ThemeProvider } from '@itwin/itwinui-react'; describe('Should have no WCAG violations', () => { Object.entries(allExamples).forEach(([name, Component]) => { it(name, () => { cy.mount( , ); cy.injectAxe({ axeCorePath: Cypress.env('axeCorePath'), }); cy.checkA11y(undefined, undefined, (violations) => { const violationData = violations.map( ({ id, impact, description, nodes }) => ({ id, impact, description, nodes: nodes.length, }), ); cy.task('table', violationData); }); }); }); }); ```

package.json

```json { "name": "a11y", "version": "0.0.0", "private": true, "type": "module", "scripts": { "test": "cypress run --component", "open": "cypress open --component", "clean": "rimraf .turbo && rimraf node_modules" }, "dependencies": { "@itwin/itwinui-react": "3.0.0-dev.3", "axe-core": "^4.7.2", "cypress": "^12.14.0", "cypress-axe": "^1.4.0", "examples": "*" } } ```

---

To run all tests headlessly: yarn test --filter=a11y from root.

To run interactively inside browser: yarn workspace a11y open.

[xman343]

Added Mayank's code to repo, a11y testing the examples now works

[mayank99]

I think we should run these tests in CI as well.

You can add a new job in build.yml that looks something like this:

  a11y:
    name: Test for a11y violations
    needs: install
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3

      - name: Use Node 18.X
        uses: actions/setup-node@v3
        with:
          node-version: 18.x

      - name: Cache node_modules
        uses: actions/cache@v3
        id: cache-node-modules
        with:
          path: |
            node_modules
            packages/*/node_modules
            apps/*/node_modules
            playgrounds/*/node_modules
          key: modules-${{ runner.os }}-${{ hashFiles('yarn.lock') }}

      - run: yarn test --filter=a11y

[xman343]

I think we should run these tests in CI as well.

You can add a new job in build.yml that looks something like this:

  a11y:
    name: Test for a11y violations
    needs: install
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3

      - name: Use Node 18.X
        uses: actions/setup-node@v3
        with:
          node-version: 18.x

      - name: Cache node_modules
        uses: actions/cache@v3
        id: cache-node-modules
        with:
          path: |
            node_modules
            packages/*/node_modules
            apps/*/node_modules
            playgrounds/*/node_modules
          key: modules-${{ runner.os }}-${{ hashFiles('yarn.lock') }}

      - run: yarn test --filter=a11y

Added!

[xman343]

Testing/playing around with this PR. If I understand correctly, we are running a pre-written script axe-core on all of our components? I am able to run the test, and it would tell me there are components that fail accessibility ~but doesn't specify how. Does it simply test for aria-label on all components?~ nevermind I just saw the exact test rules and descriptions! @xman343 @mayank99

Anything I should add, or does the code look good to you?

[LostABike]

Testing/playing around with this PR. If I understand correctly, we are running a pre-written script axe-core on all of our components? I am able to run the test, and it would tell me there are components that fail accessibility ~but doesn't specify how. Does it simply test for aria-label on all components?~ nevermind I just saw the exact test rules and descriptions! @xman343 @mayank99

Anything I should add, or does the code look good to you?

I'm not too well-versed in configurating modules and scripts so I'm just learning from your code :D But I have read them and I think they look good. Also tested functionality and seems to be working as expected 👍