storybookjs / test-runner

🚕 Turn stories into executable tests
https://storybook.js.org/docs/writing-tests/interaction-testing
MIT License
229 stars 72 forks source link

[bug] Web components play functions always return false positive #448

Open sarjir opened 5 months ago

sarjir commented 5 months ago

Describe the bug When I run play tests / interaction test with test-runner, they always pass even when they should not. The interactions part of the storybook however works as expected.

To Reproduce Steps to reproduce the behavior:

  1. Go to https://github.com/sarjir/repro-test-runner-false-positives and clone or fork the repo
  2. Install deps with pnpm
  3. Run storybook via pnpm storybook
  4. Go to the default example for the Card component. See that the interaction test fails
  5. Run pnpm run test-storybook and press a to run all tests (If it prompts you to install browsers via playwright, please do so)
  6. See that all tests pass, even when expect(true).toBe(false) definitely shouldn't

Expected behavior I would have expected my test to fail in the terminal

Screenshots Test doing what is expected in storybook interaction tab: image

Test not working in terminal: image

System System: OS: macOS 14.3.1 CPU: (10) arm64 Apple M2 Pro Shell: 5.9 - /bin/zsh Binaries: Node: 21.4.0 - ~/.nvm/versions/node/v21.4.0/bin/node Yarn: 1.22.21 - /opt/homebrew/bin/yarn npm: 10.2.4 - ~/.nvm/versions/node/v21.4.0/bin/npm pnpm: 8.15.6 - ~/.nvm/versions/node/v21.4.0/bin/pnpm <----- active Browsers: Chrome: 123.0.6312.122 Safari: 17.3.1 npmPackages: @storybook/addon-a11y: ^8.0.5 => 8.0.6 @storybook/addon-essentials: ^8.0.5 => 8.0.6 @storybook/addon-interactions: ^8.0.5 => 8.0.6 @storybook/addon-links: ^8.0.5 => 8.0.6 @storybook/addon-mdx-gfm: ^8.0.5 => 8.0.6 @storybook/blocks: ^8.0.5 => 8.0.6 @storybook/jest: ^0.2.3 => 0.2.3 @storybook/test: ^8.0.6 => 8.0.6 @storybook/test-runner: ^0.17.0 => 0.17.0 @storybook/testing-library: ^0.2.2 => 0.2.2 @storybook/web-components: ^8.0.5 => 8.0.6 @storybook/web-components-vite: ^8.0.5 => 8.0.6 storybook: ^8.0.5 => 8.0.6

Additional context Following the same steps from storybook's docs for react makes the same test fail so I have a hunch this has to do with web components.