storybookjs / test-runner

🚕 Turn stories into executable tests
https://storybook.js.org/docs/writing-tests/interaction-testing
MIT License
229 stars 72 forks source link

[bug] getStoryContext() does not include all the story tags as defined in the index.json file #452

Closed rgylesbedford closed 4 months ago

rgylesbedford commented 5 months ago

Describe the bug

I am attempting to use the tags to opt-out of specific tests within the test runner. The tags array on the context object returned from getStoryContext() does not match the tags as defined in the index.json file.

To Reproduce

  1. Storybook is built using: storybook build --test
  2. Test Runner is run with: test-storybook --junit --failOnConsole --ci --url http://127.0.0.1:6006
  3. Using the following simplified ./storybook/test-runner.ts file; the tags array on the storyContext object is always ["story"].
    
    import type { TestRunnerConfig } from "@storybook/test-runner";
    import { getStoryContext, waitForPageReady } from "@storybook/test-runner";
    import { injectAxe, checkA11y } from "axe-playwright";

// See https://storybook.js.org/docs/writing-tests/test-runner#test-hook-api const config: TestRunnerConfig = { tags: { skip: ["skip-test"], }, async preVisit(page) { await injectAxe(page); }, async postVisit(page, context) { // See https://storybook.js.org/docs/writing-tests/test-runner#helpers const storyContext = await getStoryContext(page, context);

const skipA11y = storyContext.tags.includes("skip-a11y");
const skipVisual = storyContext.tags.includes("skip-visual");

// See https://storybook.js.org/docs/writing-tests/storyshots-migration-guide#run-image-snapshot-tests-with-the-test-runner
if(!skipVisual) {
  // removed for simplicity
}

// See https://storybook.js.org/docs/writing-tests/accessibility-testing#disable-a11y-tests-with-the-test-runner
if(!skipA11y) {
  await checkA11y(page, '#storybook-root', {
    detailedReport: true,
    detailedReportOptions: {
      html: true,
    },
  });
}

} };

export default config;


**Expected behavior**

The tags array on storyContext object matches the tags listed in the index.json file for the given story.

**System**

Storybook Environment Info:

System: OS: Windows 11 10.0.22621 CPU: (16) x64 11th Gen Intel(R) Core(TM) i9-11950H @ 2.60GHz Binaries: Node: 18.20.1 - C:\Program Files\nodejs\node.EXE npm: 10.5.2 - C:\Program Files\nodejs\npm.CMD <----- active Browsers: Edge: Chromium (123.0.2420.97) npmPackages: @storybook/addon-a11y: ^8.0.9 => 8.0.9 @storybook/addon-actions: ^8.0.9 => 8.0.9 @storybook/addon-essentials: ^8.0.9 => 8.0.9 @storybook/addon-interactions: ^8.0.9 => 8.0.9 @storybook/addon-links: ^8.0.9 => 8.0.9 @storybook/addon-storysource: ^8.0.9 => 8.0.9 @storybook/blocks: ^8.0.9 => 8.0.9 @storybook/preview-api: ^8.0.9 => 8.0.9 @storybook/react: ^8.0.9 => 8.0.9 @storybook/react-vite: ^8.0.9 => 8.0.9 @storybook/test: ^8.0.9 => 8.0.9 @storybook/test-runner: ^0.17.0 => 0.17.0 eslint-plugin-storybook: ^0.8.0 => 0.8.0 storybook: ^8.0.9 => 8.0.9 storybook-dark-mode: ^4.0.1 => 4.0.1


**Additional context**

Example snippet from the index.json file:
```json
{
  "v": 4,
  "entries": {
    "examplecomponent--default": {
      "type": "story",
      "id": "examplecomponent--default",
      "name": "Default",
      "title": "ExampleComponent",
      "importPath": "./src/components/ExampleComponent/ExampleComponent.stories.tsx",
      "tags": ["skip-test", "story"]
    },
    "othercomponent--default": {
      "type": "story",
      "id": "othercomponent--default",
      "name": "Default",
      "title": "OtherComponent",
      "importPath": "./src/components/OtherComponent/OtherComponent.stories.tsx",
      "tags": ["skip-a11y", "story"]
    }
  }
}
shilman commented 4 months ago

Tag handling changed dramatically in 8.1.0, so please give it another try int he latest version and let me know what you find.

shilman commented 4 months ago

Also it's possible some changes are needed in the test runner as well @yannbf https://github.com/storybookjs/test-runner/issues/453

rgylesbedford commented 4 months ago

Tag handling changed dramatically in 8.1.0, so please give it another try in the latest version and let me know what you find.

Thanks. I got to testing the latest storybook this week and can confirm this is now working for me.

Storybook Environment Info:

  System:
    OS: Windows 11 10.0.22621
    CPU: (16) x64 11th Gen Intel(R) Core(TM) i9-11950H @ 2.60GHz
  Binaries:
    Node: 20.13.0 - C:\Program Files\nodejs\node.EXE
    npm: 10.5.2 - C:\Program Files\nodejs\npm.CMD <----- active
  Browsers:
    Edge: Chromium (125.0.2535.79)
  npmPackages:
    @storybook/addon-a11y: ^8.1.5 => 8.1.5
    @storybook/addon-actions: ^8.1.5 => 8.1.5
    @storybook/addon-essentials: ^8.1.5 => 8.1.5
    @storybook/addon-interactions: ^8.1.5 => 8.1.5
    @storybook/addon-links: ^8.1.5 => 8.1.5
    @storybook/addon-storysource: ^8.1.5 => 8.1.5
    @storybook/blocks: ^8.1.5 => 8.1.5
    @storybook/preview-api: ^8.1.5 => 8.1.5
    @storybook/react: ^8.1.5 => 8.1.5
    @storybook/react-vite: ^8.1.5 => 8.1.5
    @storybook/test: ^8.1.5 => 8.1.5
    @storybook/test-runner: ^0.18.2 => 0.18.2
    eslint-plugin-storybook: ^0.8.0 => 0.8.0
    storybook: ^8.1.5 => 8.1.5
    storybook-dark-mode: ^4.0.1 => 4.0.1