[bug] Test runner crashes with a11y tests on Storybook 8

[AmirTugi]

Describe the bug A11y tests crash in the test runner when upgrading Storybook to v8. I followed the upgrade manual and everything passes but the a11t tests in the test runner. It's saying

page.evaluate: Target crashed 
    at waitForPageReady (/mnt/ramdisk/repo/node_modules/@storybook/test-runner/dist/index.js:15314:14)
    at preVisit (/mnt/ramdisk/repo/devtools/storybook/.storybook/test-runner.ts:15:5)

To Reproduce Steps to reproduce the behavior:

Upgrade Storybook to 8.0.8 and use @storybook/test-runner@0.17.0

Expected behavior The a11y tests should pass as before

Screenshots

System

Storybook Environment Info:

  System:
    OS: macOS 14.4.1
    CPU: (12) arm64 Apple M2 Pro
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 20.10.0 - ~/.volta/tools/image/node/20.10.0/bin/node
    Yarn: 1.22.19 - ~/.volta/tools/image/yarn/1.22.19/bin/yarn <----- active
    npm: 10.2.3 - ~/.volta/tools/image/node/20.10.0/bin/npm
  Browsers:
    Safari: 17.4.1

Additional Context Here's the test-runner.ts file I have

import type { TestRunnerConfig } from '@storybook/test-runner';
import { getStoryContext } from '@storybook/test-runner';

import { checkA11y, injectAxe, configureAxe } from 'axe-playwright';

/*
 * See https://storybook.js.org/docs/writing-tests/test-runner#test-hook-api
 * to learn more about the test-runner hooks API.
 */
const config: TestRunnerConfig = {
  tags: {
    include: ['a11y-include'],
  },
  async preVisit(page) {
    await injectAxe(page);
  },
  async postVisit(page, context) {
    // Get the entire context of a story, including parameters, args, argTypes, etc.
    const storyContext = await getStoryContext(page, context);

    // Apply story-level a11y rules
    await configureAxe(page, {
      rules: storyContext.parameters?.['a11y']?.config?.rules,
    });

    await checkA11y(
      page,
      '#storybook-root',
      {
        detailedReport: true,
        axeOptions: {
          // Limit the result types to only violations that the test managed to find
          resultTypes: ['violations'],
        },
        detailedReportOptions: {
          html: true,
        },
      },
      false,
      'v2'
    );
  },
};

export default config;

[ethriel3695]

Hi @AmirTugi, we will take a look and let you know what we find.

[yannbf]

Hey @AmirTugi thanks a lot for elaborating, but could you please provide a reproduction repo? You can use https://storybook.new for a minimal repro setting. I tried the test-runner with Storybook 8.0.8 with your test-runner.ts file and the a11y tests worked without issues.

I actually find it weird that your tests failed with waitForPageReady while in your snippet such function is not used. That function is not used internally by the test-runner, only if you add it yourself. I think you might have it in your preVisit hook. Maybe what's going on is you are using Webpack and somehow the waitForPageReady function is hanging because of some Webpack request that won't resolve, but I am not sure as I don't know the configuration of your project.

[AmirTugi]

Managed to fix the issue. Seems like it relied in axe-playwright which is what the official Storybook docs suggest.

I replaced it with @axe-core/playwright and configured it almost the same as the axe-playwright, and now it works.

This is my new set-up.

import type { TestRunnerConfig } from '@storybook/test-runner';
import { getStoryContext } from '@storybook/test-runner';
import { AxeBuilder } from '@axe-core/playwright';

/*
 * See https://storybook.js.org/docs/writing-tests/test-runner#test-hook-api
 * to learn more about the test-runner hooks API.
 */
const config: TestRunnerConfig = {
  tags: {
    include: ['a11y-include'],
    skip: ['a11y-skip'],
  },
  async postVisit(page, context) {
    const storyContext = await getStoryContext(page, context);
    const disabledRules = (storyContext.parameters?.['a11y']?.config?.rules ?? [])
      .filter((rule: { enabled: boolean }) => !rule.enabled)
      .map((rule: { id: string }) => rule.id)

    const results = await new AxeBuilder({ page })
      .include('#storybook-root')
      .options({
        // Limit the result types to only violations that the test managed to find
        resultTypes: ['violations'],
      })
      .withTags(['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa'])
      .disableRules(disabledRules)
      .analyze()

    expect(results.violations).toEqual([]);
  },
};

export default config;

The downside of this is that the error reporting is the expect().toEqual([]) which is less readable than the reporter from axe-playwright.

But I'm sure that with a bit of effort it can be fixed.