storybookjs / test-runner

🚕 Turn stories into executable tests
https://storybook.js.org/docs/writing-tests/interaction-testing
MIT License
229 stars 72 forks source link

[Bug]: Include/Exclude tags dont behave as expected #470

Open sw-tracker opened 4 months ago

sw-tracker commented 4 months ago

Describe the bug

I have created 1 tag which I have added to some tests.

tags: ['sequential-test-execution'],

I then run these 2 commands:

npx test-storybook --browsers chromium --coverage --junit --ci --maxWorkers=1 --includeTags="sequential-test-execution"
npx test-storybook --browsers chromium --coverage --junit --ci --maxWorkers=1 --excludeTags="sequential-test-execution"

I expect the command that has includeTags to run my 24 tests and the command that has excludeTags to run all of the other tests that dont have the tag sequential-test-execution. But instead I get this:

image image

I have 24 story files in total. As you can see, the first command is skipping all tests (even those that have the tag), and the second command is executing all tests. How am I meant to use this tag to filter tests using the test-runner?

To Reproduce

import { StoryFn, StoryObj } from '@storybook/react';

export default {
  title: 'Runtime-Integration/MyModal',
  tags: ['sequential-test-execution'],
  component: MyModalWC,
  parameters: {
    docs: componentDocs,
  },
  argTypes: {
    ...wcPropsDescriptions,
  },
} as StoryObj<typeof MyModalWC>;

System

Environment Info:

  System:
    OS: macOS 14.3
    CPU: (10) arm64 Apple M2 Pro
  Binaries:
    Node: 18.20.2 - ~/.local/share/mise/installs/node/18.20.2/bin/node
    Yarn: 1.22.22 - /opt/homebrew/bin/yarn
    npm: 10.5.0 - ~/.local/share/mise/installs/node/18.20.2/bin/npm
  Browsers:
    Chrome: 125.0.6422.77
    Edge: 125.0.2535.51
    Safari: 17.3


### Additional context

_No response_
yannbf commented 4 months ago

Hey @sw-tracker can you share a reproduction repo? I just tried and I got the correct behavior for include:

image

And exclude:

image
sw-tracker commented 4 months ago

I will setup a repo to reproduce it. In the meantime I can give you the dependency versions I have in my project:

    "@storybook/addon-actions": "7.0.2",
    "@storybook/addon-coverage": "0.0.8",
    "@storybook/addon-docs": "7.0.2",
    "@storybook/addon-essentials": "7.0.2",
    "@storybook/addon-interactions": "7.0.2",
    "@storybook/addon-links": "7.0.2",
    "@storybook/jest": "0.1.0",
    "@storybook/react": "7.0.2",
    "@storybook/react-webpack5": "7.0.2",
    "@storybook/test-runner": "0.18.1",
    "@storybook/testing-library": "0.1.0",
    "storybook": "7.0.2",
    "storybook-addon-manual-mocks": "0.1.1",
    "storybook-addon-mock": "4.0.0",

Maybe it has something to do with the versions I have.

sw-tracker commented 4 months ago

I found how to reproduce it, as soon as you have playwright tests in the stories file, the tags start to misbehave.

BlahButton2PrimaryStory2.play = async ({ args }) => {
};
sw-tracker commented 4 months ago

You can see the bug here: https://github.com/sw-tracker/storybook-reproduce-tag-problem

image

The command above should only run 1 of the tests, not both.