storybookjs / storybook

Storybook is the industry standard workshop for building, documenting, and testing UI components in isolation
https://storybook.js.org
MIT License
83.77k stars 9.18k forks source link

[Bug]: Interaction tests using RHF fail when configuring playwright in headed mode with devtools #24719

Open kpericin opened 9 months ago

kpericin commented 9 months ago

Describe the bug

When using @storybook/test-runner to run interaction test for simple form managed by react-hook-form, tests will randomly fail a few times in a hundred runs. At first I thought it is caused by https://github.com/storybookjs/storybook/issues/18258, however, after upgrading everything the issue persisted.

After a lot of trial and error, I managed to get to a consistent reproduction which can be achieved by configuring test runner to run in headed mode with devtools enabled. In that configuration, simple test will always fail. Text gets entered in the input field but empty string gets submitted.

I'm not entirely sure the issue is in storybook. I'm assuming it is because I cannot reproduce it if I test the same storybook page using playwright directly. Or, if I run vite dev server and point playwright tests to it.

To Reproduce

Repository of the reproduction can be found here.

To reproduce, first install dependencies then start storybook dev server. Open http://localhost:6009/?path=/story/nativerhfinteraction--text-input, it will most likely work when run in storybook directly. Now run test-storybook. It will open Chromium browser with devtools enabled and you'll see toast notification in the bottom left corner which will have empty text.

Next, try running test script. It will also open Chromium with devtools enabled, but this time the test will pass because notification will have the correct text.

System

Storybook Environment Info:

  System:
    OS: macOS 14.1
    CPU: (12) arm64 Apple M2 Max
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 18.16.0 - /usr/local/bin/node
    Yarn: 1.22.19 - /usr/local/bin/yarn
    npm: 10.1.0 - /opt/homebrew/bin/npm
    pnpm: 8.8.0 - /usr/local/bin/pnpm <----- active
  Browsers:
    Chrome: 119.0.6045.105
    Safari: 17.1
  npmPackages:
    @storybook/addon-essentials: 7.5.2 => 7.5.2 
    @storybook/addon-interactions: 7.5.2 => 7.5.2 
    @storybook/addon-links: 7.5.2 => 7.5.2 
    @storybook/blocks: 7.5.2 => 7.5.2 
    @storybook/jest: 0.2.2 => 0.2.2 
    @storybook/react: 7.5.2 => 7.5.2 
    @storybook/react-vite: 7.5.2 => 7.5.2 
    @storybook/test-runner: 0.13.0 => 0.13.0 
    @storybook/testing-library: 0.2.2 => 0.2.2 
    storybook: ^7.5.2 => 7.5.2

Additional context

No response

kpericin commented 6 months ago

I added another case, using TanStack form: https://github.com/kpericin/storybook-play-function-repro/blob/main/src/stories/TanstackFormInteraction.stories.js

It has the same issue when using test-storybook - text gets entered in the input field but empty string gets submitted.