storybookjs / test-runner

🚕 Turn stories into executable tests
https://storybook.js.org/docs/writing-tests/interaction-testing
MIT License
233 stars 72 forks source link

[Bug]: how to debug test runner network issues #480

Closed kraftman closed 5 months ago

kraftman commented 5 months ago

Describe the bug

I have about 100 interaction tests that make a single network request, which is mocked out using msw when running locally and in PRs. I want to remove the mock when merging to master to e2e test the stories against our QA API. When I remove the mock, all of the tests pass locally in the UI, and they all pass individually when run in the test runner, however when run together, 1 or 2 will fail. I believe that occasionally the network request is failing or taking too long, but its very hard to debug. Because of CORS issues I have to run chromium so that I can use '--disable-web-security', and then use 'page.on('console', (err) => {' to get it to log the browser console, however since there are just over 100 tests its pretty hard to dig into whats going on, and I cant just narrow it down to one test. I thought I could just set the flakey test to retry, but I havent found a way to tell the test runner to retry, is that possible? Another idea was to just set the mock service worker to act as a spy and not a mock, and log out any failed responses, but it seems that the only way to do that is to use passthrough(), and that doesnt give access to the response. I tried manually intercepting the response in MSW and then sending a fetch to the API based on the request, but that seemed to break MSW completely with no logs. Any other ideas for workarounds for this? Can I slow the tests down or batch them to avoid overloading the API? Is there another way to inspect the network requests to confirm that it is infact failing, or being too slow? thanks.

To Reproduce

Create 100+ tests that make a network request to a potentially flaky API?

System

System:
    OS: Linux 5.15 Ubuntu 22.04.2 LTS 22.04.2 LTS (Jammy Jellyfish)
    CPU: (24) x64 AMD Ryzen 9 7900X 12-Core Processor
    Shell: 5.8.1 - /usr/bin/zsh
  Binaries:
    Node: 20.13.0 - ~/.nvm/versions/node/v20.13.0/bin/node
    Yarn: 1.22.22 - ~/mason-web/node_modules/.bin/yarn <----- active
    npm: 10.5.2 - ~/.nvm/versions/node/v20.13.0/bin/npm
  npmPackages:
    @storybook/addon-actions: ^8.0.10 => 8.0.10
    @storybook/addon-controls: ^8.0.10 => 8.0.10
    @storybook/addon-designs: ^8.0.1 => 8.0.1
    @storybook/addon-essentials: ^8.0.10 => 8.0.10
    @storybook/addon-interactions: ^8.0.10 => 8.0.10
    @storybook/addon-links: ^8.0.10 => 8.0.10
    @storybook/addon-storysource: ^8.0.10 => 8.0.10
    @storybook/addon-viewport: ^8.0.10 => 8.0.10
    @storybook/addon-webpack5-compiler-babel: ^3.0.3 => 3.0.3
    @storybook/builder-webpack5: ^8.1.1 => 8.1.1
    @storybook/manager-api: ^8.0.10 => 8.0.10
    @storybook/react: ^8.0.10 => 8.0.10
    @storybook/react-webpack5: ^8.0.10 => 8.0.10
    @storybook/source-loader: ^8.0.10 => 8.0.10
    @storybook/test: ^8.0.10 => 8.0.10
    @storybook/test-runner: ^0.18.0 => 0.18.0
    msw-storybook-addon: ^2.0.2 => 2.0.2
    storybook: ^8.0.10 => 8.0.10

Additional context

No response

kraftman commented 5 months ago

eventually solved this by setting all of the timeouts much higher, 15-20 seconds