search

storybookjs / test-runner

🚕 Turn stories into executable tests
https://storybook.js.org/docs/writing-tests/interaction-testing
MIT License
231 stars 72 forks source link

[Bug] Runner hang randomly in middle of the tests #326

Open Hideman85 opened 1 year ago

Hideman85 commented 1 year ago

We are running 270 tests in CICD and quite often now the runner hangs randomly in the middle.

The chrome instances spawned are looking like this:

.yarn/cache/pw-browsers/chromium-1048/chrome-linux/chrome --disable-field-trial-config --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disable-background-timer-throttling --disable-backgrounding-occluded-windows --disable-back-forward-cache --disable-breakpad --disable-client-side-phishing-detection --disable-component-extensions-with-background-pages --disable-component-update --no-default-browser-check --disable-default-apps --disable-dev-shm-usage --disable-extensions --disable-features=ImprovedCookieControls,LazyFrameLoading,GlobalMediaControls,DestroyProfileOnBrowserClose,MediaRouter,DialMediaRouteProvider,AcceptCHFrame,AutoExpandDetailsElement,CertificateTransparencyComponentUpdater,AvoidUnnecessaryBeforeUnloadCheckSync,Translate --allow-pre-commit-input --disable-hang-monitor --disable-ipc-flooding-protection --disable-popup-blocking --disable-prompt-on-repost --disable-renderer-backgrounding --disable-sync --force-color-profile=srgb --metrics-recording-only --no-first-run --enable-automation --password-store=basic --use-mock-keychain --no-service-autorun --export-tagged-pdf --headless --hide-scrollbars --mute-audio --blink-settings=primaryHoverType=2,availableHoverTypes=2,primaryPointerType=4,availablePointerTypes=4 --no-sandbox --allow-running-insecure-content --autoplay-policy=user-gesture-required --disable-domain-reliability --disable-print-preview --disable-setuid-sandbox --disable-site-isolation-trials --ignore-gpu-blocklist --in-process-gpu --no-pings --no-zygote --disable-gpu --user-data-dir=/tmp/playwright_chromiumdev_profile-qY9qLO --remote-debugging-pipe --no-startup-window
.yarn/cache/pw-browsers/chromium-1048/chrome-linux/chrome --type=utility --utility-sub-type=network.mojom.NetworkService --lang=en-US --service-sandbox-type=none --no-sandbox --disable-dev-shm-usage --use-angle=swiftshader-webgl --use-gl=angle --mute-audio --headless --shared-files=v8_context_snapshot_data:100 --field-trial-handle=0,i,16082858727042721604,5801625611727073452,131072 --enable-features=NetworkService,NetworkServiceInProcess --disable-features=AcceptCHFrame,AutoExpandDetailsElement,AvoidUnnecessaryBeforeUnloadCheckSync,CertificateTransparencyComponentUpdater,DestroyProfileOnBrowserClose,DialMediaRouteProvider,GlobalMediaControls,ImprovedCookieControls,LazyFrameLoading,MediaRouter,PaintHolding,Translate

Our CICD logs shorten are looking like this:

[TEST] FAIL browser: chromium packages/settings/src/components/MyComponent1.stories.tsx
[TEST]   ● settings/src/components/MyComponent1 › MyComponent1Variant1 › smoke-test
[TEST] 
[TEST]     page.evaluate: Target crashed
[TEST] 
[TEST]       at testFn (packages/settings/src/components/MyComponent1.stories.tsx:25:37)
[TEST]       at Object.<anonymous> (packages/settings/src/components/MyComponent1.stories.tsx:46:17)

///// 20min of nothing only the ram usage logger gives us no memory issue at any time
               total        used        free      shared  buff/cache   available
Mem:           7.6Gi       2.6Gi       2.3Gi       0.0Ki       2.7Gi       4.7Gi
Swap:          4.0Gi        48Mi       4.0Gi

///// Here I manually kill chrome from the CICD console
///// => therefore quite some tests are instant failing

[TEST] FAIL browser: chromium packages/app/src/components/MyComponent2.stories.tsx
[TEST]   ● Test suite failed to run
[TEST] 
[TEST]     page.exposeBinding: Browser has been closed
[TEST] 
[TEST]       at setupPage (node_modules/@storybook/test-runner/dist/cjs/setup-page.js:87:14)
[TEST]       at CustomEnvironment.setup (node_modules/@storybook/test-runner/playwright/custom-environment.js:9:5)
[TEST] 
[TEST] FAIL browser: chromium packages/app/src/components/MyComponent3.stories.tsx
[TEST]   ● Test suite failed to run
[TEST] 
[TEST]     TypeError: Cannot read properties of undefined (reading 'pages')
[TEST] 
[TEST]       at CustomEnvironment.teardown (node_modules/jest-playwright-preset/lib/PlaywrightEnvironment.js:233:43)
[TEST]       at CustomEnvironment.teardown (node_modules/@storybook/test-runner/playwright/custom-environment.js:13:17)

///// Then the runner continue and doing the report

[TEST] Coverage file (8167631 bytes) written to .nyc_output/coverage.json

This is not related to a specific component, it looks to be randomly hanging. I try to kill only the second chrome process (that I guess is the tab) the first chrome spawn a new instance right away so somehow chrome is still responsive. We do not seem to reach any critical memory or cpu usage. And the error happen randomly.

Could it be the runner or playwright lost the devtool connection to chrome for some reason and do not try to reestablish it?

Any help there would be appreciated 🙏

yannbf commented 1 year ago

Hey there! Thank for opening this issue. Unfortunately I have never hit this situation, neither do I know how to reproduce it :/ Is it possible there is a navigation happening in any of your stories? If there is, that could likely be the cause of the issue.