search

ember-a11y / ember-a11y-testing

A suite of accessibility tests that can be run within the Ember testing framework
MIT License
138 stars 52 forks source link
a11y a11y-testing accessibility

readme

ember-a11y-testing

CI Build NPM Version Ember Observer Score

ember-a11y-testing is a wrapper around Deque Labs' axe-core accessibility testing engine. It integrates into your testing environment using either a one-time setup, or in individual tests using an a11yAudit() test helper.

Compatibility

Note: we enforce a peerDependency of @ember/test-helpers. If you encounter the following message:

ember-a11y-testing has the following unmet peerDependencies:
  * @ember/test-helpers: `^2.0.0`; it was resolved to `x.x.x`

please update your version of @ember/test-helpers in your package.json accordingly.

Note: Users who are using Node <= 14 in their apps should use ember-a11y-testing@^5.2.1.

Installation

ember install ember-a11y-testing

Usage

Usage of ember-a11y-testing in your tests can be done in one of two ways:

  1. A one-time setup in your tests/test-helper.js file using setupGlobalA11yHooks
  2. In individual tests using the a11yAudit test helper.

axe Options

When using the a11yAudit helper, you can pass in axe-core options that are passed to axe.run. These options are documented in the axe-core API docs. The rule definitions are documented on dequeuniversity.com/rules.

Each of the following sections individually details how to set aXe options for your tests.

setupGlobalA11yHooks Usage

The setupGlobalA11yHooks function is intended to be imported and invoked a single time in tests/test-helper.js for your entire test suite.

export interface InvocationStrategy {
  (helperName: string, label: string): boolean;
}

export interface GlobalA11yHookOptions {
  helpers: HelperName[];
}

type HelperName =
  | 'blur'
  | 'click'
  | 'doubleClick'
  | 'fillIn'
  | 'focus'
  | 'render'
  | 'scrollTo'
  | 'select'
  | 'tab'
  | 'tap'
  | 'triggerEvent'
  | 'triggerKeyEvent'
  | 'typeIn'
  | 'visit';

export const DEFAULT_A11Y_TEST_HELPER_NAMES = [
  'visit',
  'click',
  'doubleClick',
  'tap',
];

export function setupGlobalA11yHooks(
  shouldAudit: InvocationStrategy,
  audit: (...args: any[]) => PromiseLike<void> = a11yAudit,
  options: GlobalA11yHookOptions = { helpers: DEFAULT_A11Y_TEST_HELPER_NAMES }
);

The setupGlobalA11yHooks function takes three parameters:

Using a custom InvocationStrategy implementation will allow you to maintain a high level of control over your test invocations. Examples of invocation strategies can be found in this repository's tests.

To use, import and invoke the global setup function, passing in your specific invocation strategy:

// tests/test-helper.js
import Application from 'my-app/app';
import config from 'my-app/config/environment';
import { setApplication } from '@ember/test-helpers';
import { start } from 'ember-qunit';
import { setupGlobalA11yHooks } from 'ember-a11y-testing/test-support';
setApplication(Application.create(config.APP));

setupGlobalA11yHooks(() => true);

start();

:warning: It's important to note that you must also use the enableA11yAudit query parameter in order to force audits. This setting is required in addition to any invocation strategy you provide.

By default, audits will be run on visit, click, doubleClick, and tap. To add additional helpers to hook into, specify them by name in the options.helpers argument. Note that this option specifies the complete set of helpers to hook into; to include the defaults you must import them and splat them into the array as shown below.

import {
  setupGlobalA11yHooks,
  DEFAULT_A11Y_TEST_HELPER_NAMES,
} from 'ember-a11y-testing/test-support';

setupGlobalA11yHooks(() => true, {
  helpers: [...DEFAULT_A11Y_TEST_HELPER_NAMES, 'render', 'tab'],
});

Setting Options using setRunOptions

You can provide options to axe-core for your tests using the setRunOptions API. This API is helpful if you don't have access to the a11yAudit calls directly, such as when using the setupGlobalA11yHooks, or if you want to set the same options for all tests in a module.

Options can be set a few ways:

Globally:

// tests/test-helper.js
import { setRunOptions } from 'ember-a11y-testing/test-support';

setRunOptions({
  rules: {
    region: { enabled: true },
  },
});

Test module level:

import { module, test } from 'qunit';
import { setRunOptions } from 'ember-a11y-testing/test-support';

module('some test module', function (hooks) {
  hooks.beforeEach(function () {
    setRunOptions({
      rules: {
        region: { enabled: true },
      },
    });
  });

  // ...
});

Individual test level:

import { module, test } from 'qunit';
import { a11yAudit, setRunOptions } from 'ember-a11y-testing/test-support';

module('some test module', function (hooks) {
  test('some test', function (assert) {
    setRunOptions({
      rules: {
        region: { enabled: true },
      },
    });

    // ...
    a11yAudit();
    // ...
  });

  // ...
});

When using setRunOptions during a test, the options you set are automatically reset when the test completes.

a11yAudit Usage

ember-a11y-testing provides a test helper to run accessibility audits on specific tests within your test suite. The a11yAudit helper is an async test helper which can be used in a similar fashion to other @ember/test-helpers helpers:

In Application tests:

import { visit } from '@ember/test-helpers';
import { a11yAudit } from 'ember-a11y-testing/test-support';

module('Some module', function () {
  //...

  test('Some test case', async function (assert) {
    await visit('/');
    await a11yAudit();
    assert.ok(true, 'no a11y errors found!');
  });
});

The helper is also able to be used in Integration/Unit tests like so:

import { render } from '@ember/test-helpers';
import { a11yAudit } from 'ember-a11y-testing/test-support';

// ...elided for brevity

test('Some test case', function (assert) {
  await render(hbs`{{some-component}}`);

  let axeOptions = {
    rules: {
      'button-name': {
        enabled: false,
      },
    },
  };

  await a11yAudit(this.element, axeOptions);

  assert.ok(true, 'no a11y errors found!');
});

Setting Options with a11yAudit

The helper can optionally accept a "context" on which to focus the audit as either a selector string or an HTML element. You can also provide a secondary parameter to specify axe-core options:

test('Some test case', async function (assert) {
  let axeOptions = {
    rules: {
      'button-name': {
        enabled: false,
      },
    },
  };

  await visit('/');
  await a11yAudit(axeOptions);

  assert.ok(true, 'no a11y errors found!');
});

Or specify options as a single argument:

test('Some test case', async function (assert) {
  let axeOptions = {
    rules: {
      'button-name': {
        enabled: false,
      },
    },
  };

  await visit('/');
  await a11yAudit('.modal', axeOptions);

  assert.ok(true, 'no a11y errors found!');
});

Force Running audits

ember-a11y-testing allows you to force audits if enableA11yAudit is set as a query param on the test page or the ENABLE_A11Y_AUDIT environment variable is provided. This is useful if you want to conditionally run accessibility audits, such as during nightly build jobs.

To do so, import and use shouldForceAudit from ember-a11y-testing, as shown below.

// `&enableA11yAudit` set in the URL
import { a11yAudit, shouldForceAudit } from 'ember-a11y-testing/test-support';

test(
  'Some test case',
  await function (assert) {
    await visit('/');

    if (shouldForceAudit()) {
      await a11yAudit();
    }
    assert.ok(true, 'no a11y errors found!');
  }
);
// No `enableA11yAudit` set in the URL
import { a11yAudit, shouldForceAudit } from 'ember-a11y-testing/test-support';

test(
  'Some test case',
  await function (assert) {
    await visit('/');

    if (shouldForceAudit()) {
      await a11yAudit(); // will not run
    }
    // ...
  }
);

You can also create your own app-level helper, which will conditionally check whether to run the audits or not:

export function a11yAuditIf(contextSelector, axeOptions) {
  if (shouldForceAudit()) {
    return a11yAudit(contextSelector, axeOptions);
  }

  return resolve(undefined, 'a11y audit not run');
}

QUnit and Testem integration

You can setup a new configuration checkbox in QUnit and Testem by using the setupQUnitA11yAuditToggle. When the checkbox is checked, it will set enableA11yAudit as a query param.

To use, import and invoke the setup function, passing in your QUnit instance:

// tests/test-helper.js
import Application from 'my-app/app';
import config from 'my-app/config/environment';
import * as QUnit from 'qunit';
import { setApplication } from '@ember/test-helpers';
import { start } from 'ember-qunit';
import { setupGlobalA11yHooks, setupQUnitA11yAuditToggle } from 'ember-a11y-testing/test-support';
setApplication(Application.create(config.APP));

setupGlobalA11yHooks(() => true);
setupQUnitA11yAuditToggle(QUnit);

start();

Logging violations to the console

This addon provides the capability of summarizing all violations found during tests, and outputting those failures to the console once the test suite is completed. To enable this functionality, import setupConsoleLogger and invoke in your tests/test-helper.js file:

import Application from 'my-app/app';
import config from 'my-app/config/environment';
import { setApplication } from '@ember/test-helpers';
import { start } from 'ember-qunit';
import { setupConsoleLogger } from 'ember-a11y-testing/test-support';

setApplication(Application.create(config.APP));

setupConsoleLogger();

start();

Example:

Console Logger Output Example

Test Middleware

This addon provides middleware - code that allows the browser to talk to the node process running the tests via testem. This is useful in scenarios such as internal compliance monitoring used to track accessibility grades.

The middleware reporter writes the results containing all violations detected in all tests to a JSON file stored in a directory, ember-a11y-report, in your application or addon's root directory.

:warning: Audit report files get generated in an additive manner, typically resulting in the a11y-audit-report directory growing in size as subsequent test suites are run. Environments with specific space size restrictions will require an explicit strategy to manage the deletion of older reports, as this addon no longer does so.

To use the middleware reporter, import setupMiddlewareReporter and invoke in your tests/test-helper.js file:

import Application from 'my-app/app';
import config from 'my-app/config/environment';
import { setApplication } from '@ember/test-helpers';
import { start } from 'ember-qunit';
import { setupMiddlewareReporter } from 'ember-a11y-testing/test-support';

setApplication(Application.create(config.APP));

setupMiddlewareReporter();

start();

A helper function is available to use the middleware reporter conditionally, allowing interoperability between the default reporter and the middleware reporter. Import useMiddlewareReporter and apply as a check around the setupMiddlewareReporter function in tests/test-helper.js. The middleware reporter will now only be invoked when enableA11yMiddlewareReporter is set as a query param on the test page or the ENABLE_A11Y_MIDDLEWARE_REPORTER environment variable is provided.

import Application from 'my-app/app';
import config from 'my-app/config/environment';
import { setApplication } from '@ember/test-helpers';
import { start } from 'ember-qunit';
import {
  setupMiddlewareReporter,
  useMiddlewareReporter,
} from 'ember-a11y-testing/test-support';

setApplication(Application.create(config.APP));

if (useMiddlewareReporter()) {
  // Only runs if `enableA11yMiddlewareReporter` is set in URL
  setupMiddlewareReporter();
}

start();

Note, as a convenience, useMiddlewareReporter automatically forces audits, thus explicitly specifying the enableA11yAudit query param or the ENABLE_A11Y_AUDIT environment variable is unnecessary.

Development Usage

While this addon previously included a number of components that would aid in identifying axe violations during development, those have been deprecated in favor of other, industry standard tools such as: