search

microsoft / playwright

Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.
https://playwright.dev
Apache License 2.0
65.72k stars 3.58k forks source link

[BUG] screenshots hang on arm64 docker image #29073

Open aristidesstaffieri opened 8 months ago

aristidesstaffieri commented 8 months ago

System info

Source code

https://github.com/stellar/freighter/pull/1084 Tests are here - https://github.com/stellar/freighter/tree/master/extension/e2e-tests

Steps docker run --platform=linux/arm64 --ipc=host --rm --network host -v $(pwd):/work/ -w /work/ -it --cap-add=SYS_ADMIN --init mcr.microsoft.com/playwright:v1.41.0-jammy /bin/bash yarn && yarn test:e2e

Expected Screenshots to be generated for linux.

Actual Runner hangs indefinitely

Info I'm able to run these tests locally on my M1, and can include the toHaveScreenshot assertions and get the expected screenshots and desired behavior. When I try to run the same tests in the docker container, the runner hangs until it times out, unable to take the screenshot. I've tried increasing my timeout by a lot, and taking partial page screenshots, as well as disabling animations. Interestingly, the very first screenshot does get taken but any subsequent ones hang until a timeout happens.

Screenshot 2024-01-19 at 1 20 16 PM
yury-s commented 7 months ago

Does it also hang without the extension?

yury-s commented 7 months ago

stellar/freighter#1084

Note that the linux screenshots are missing, so the tests are bound to fail. Can you generate the expected images and retry the tests?

Btw, on github actions the default hardware is x64, not arm and the tests are failing there too in your PR.

aristidesstaffieri commented 7 months ago

Does it also hang without the extension?

The whole project is a browser extension so I don't think I can test that with this repo, but if I write a very rudimentary test suite targeting a web app context it does not hang so I think it's fair to be suspicious of using chromium.launchPersistentContext and trying to take screenshots on that image.

aristidesstaffieri commented 7 months ago

Does it also hang without the extension?

The whole project is a browser extension so I don't think I can test that with this repo, but if I write a very rudimentary test suite targeting a web app context it does not hang so I think it's fair to be suspicious of using chromium.launchPersistentContext and trying to take screenshots on that image.

This happens when trying to generate the images for linux as well. The tests in my action are run on an ubuntu image, you can see in the output that it is expecting the images to be for linux.

aristidesstaffieri commented 7 months ago

If I run this with the debug flag, I do see this warning about the headless option even though I do have that option set to true

root@docker-desktop:/work/extension# npx playwright test --debug

Running 5 tests using 1 worker

  ✘  1 [chromium] › onboarding.test.ts:8:5 › Welcome page loads (762ms)

  1) [chromium] › onboarding.test.ts:8:5 › Welcome page loads ──────────────────────────────────────

    Error: browserType.launchPersistentContext: Target page, context or browser has been closed
    Browser logs:

    ╔════════════════════════════════════════════════════════════════════════════════════════════════╗
    ║ Looks like you launched a headed browser without having a XServer running.                     ║
    ║ Set either 'headless: true' or use 'xvfb-run <your-playwright-app>' before running Playwright. ║
    ║                                                                                                ║
    ║ <3 Playwright Team                                                                             ║
    ╚════════════════════════════════════════════════════════════════════════════════════════════════╝
    Call log:
      - <launching> /ms-playwright/chromium-1097/chrome-linux/chrome --disable-field-trial-config --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disable-background-timer-throttling --disable-backgrounding-occluded-windows --disable-back-forward-cache --disable-breakpad --disable-client-side-phishing-detection --disable-component-extensions-with-background-pages --disable-component-update --no-default-browser-check --disable-default-apps --disable-dev-shm-usage --disable-extensions --disable-features=ImprovedCookieControls,LazyFrameLoading,GlobalMediaControls,DestroyProfileOnBrowserClose,MediaRouter,DialMediaRouteProvider,AcceptCHFrame,AutoExpandDetailsElement,CertificateTransparencyComponentUpdater,AvoidUnnecessaryBeforeUnloadCheckSync,Translate,HttpsUpgrades,PaintHolding --allow-pre-commit-input --disable-hang-monitor --disable-ipc-flooding-protection --disable-popup-blocking --disable-prompt-on-repost --disable-renderer-backgrounding --force-color-profile=srgb --metrics-recording-only --no-first-run --enable-automation --password-store=basic --use-mock-keychain --no-service-autorun --export-tagged-pdf --disable-search-engine-choice-screen --no-sandbox --headless=new --disable-extensions-except=/work/extension/build --load-extension=/work/extension/build --user-data-dir=/tmp/playwright_chromiumdev_profile-mgOS8K --remote-debugging-pipe about:blank
      - <launched> pid=2145
      - [pid=2145][err] [2145:2159:0125/162140.080288:ERROR:bus.cc(407)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
      - [pid=2145][err] [2145:2162:0125/162140.099019:ERROR:bus.cc(407)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
      - [pid=2145][err] [2145:2162:0125/162140.099099:ERROR:bus.cc(407)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
      - [pid=2145][err] [2145:2159:0125/162140.107588:ERROR:bus.cc(407)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
      - [pid=2145][err] [2145:2159:0125/162140.107929:ERROR:bus.cc(407)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
      - [pid=2145][err] [2145:2159:0125/162140.108116:ERROR:bus.cc(407)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
      - [pid=2145][err] [2145:2159:0125/162140.108143:ERROR:bus.cc(407)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
      - [pid=2145][err] [2145:2145:0125/162140.121672:ERROR:policy_logger.cc(156)] :components/enterprise/browser/controller/chrome_browser_cloud_management_controller.cc(161) Cloud management controller initialization aborted as CBCM is not enabled. Please use the `--enable-chrome-browser-cloud-management` command line flag to enable it if you are not using the official Google Chrome build.
      - [pid=2145][err] [2145:2159:0125/162140.146086:ERROR:bus.cc(407)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
      - [pid=2145][err] [2145:2159:0125/162140.146119:ERROR:bus.cc(407)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
      - [pid=2145][err] [2145:2231:0125/162140.273233:ERROR:bus.cc(407)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
      - [pid=2145][err] [2145:2231:0125/162140.273364:ERROR:bus.cc(407)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
      - [pid=2145][err] [2145:2231:0125/162140.273400:ERROR:bus.cc(407)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
      - [pid=2145][err] [2145:2231:0125/162140.273474:ERROR:bus.cc(407)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
      - [pid=2145][err] [2145:2231:0125/162140.273552:ERROR:bus.cc(407)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory

       at test-fixtures.ts:10

       8 |   context: async ({}, use) => {
       9 |     const pathToExtension = path.join(__dirname, "../build");
    > 10 |     const context = await chromium.launchPersistentContext("", {
         |                     ^
      11 |       headless: true,
      12 |       args: [
      13 |         `--headless=new`,

        at Object.context (/work/extension/e2e-tests/test-fixtures.ts:10:21)

Testing stopped early after 1 maximum allowed failures.
  1 failed
    [chromium] › onboarding.test.ts:8:5 › Welcome page loads ───────────────────────────────────────
  4 did not run
yury-s commented 7 months ago

If I run this with the debug flag, I do see this warning about the headless option even though I do have that option set to true

--debug option implies --headed, this is why you see the error. You can fix it by bringing up an X server in the docker.

I tried running a subset of the tests with toHaveScreenshot uncommented and if I generate the expectations first with -u option, on the next run they will pass (ok, they seem to be flaky but that's a different problem). The command I ran are:

CI=1 yarn test:e2e --  onboarding.test.ts:8 onboarding.test.ts:17 onboarding.test.ts:59 -u

and after that one generated expected images, I ran

CI=1 yarn test:e2e --  onboarding.test.ts:8 onboarding.test.ts:17 onboarding.test.ts:59

with the following output:

# CI=1 yarn test:e2e --  onboarding.test.ts:8 onboarding.test.ts:17 onboarding.test.ts:59
yarn run v1.22.21                                                                                                                                            
warning From Yarn 1.0 onwards, scripts don't require "--" for options to be forwarded. In a future version, any explicit "--" will be forwarded as-is to the scripts.
$ yarn workspace extension test:e2e onboarding.test.ts:8 onboarding.test.ts:17 onboarding.test.ts:59                                                         
$ playwright test onboarding.test.ts:8 onboarding.test.ts:17 onboarding.test.ts:59                                                                           

Running 3 tests using 1 worker

  ✓  1 [chromium] › onboarding.test.ts:8:5 › Welcome page loads (4.9s)
  ✓  2 [chromium] › onboarding.test.ts:17:5 › Create new wallet (4.6s)
  ✘  3 [chromium] › onboarding.test.ts:59:5 › Import 12 word wallet (9.8s)
  ✓  4 [chromium] › onboarding.test.ts:59:5 › Import 12 word wallet (retry #1) (4.5s)

  1) [chromium] › onboarding.test.ts:59:5 › Import 12 word wallet ──────────────────────────────────

    Error: Screenshot comparison failed:

      Timeout 5000ms exceeded.

    Expected: /workspaces/playwright-browsers/freighter/extension/e2e-tests/onboarding.test.ts-snapshots/wallet-import-complete-page-chromium-linux.png

    Call log:
      - page._expectScreenshot with timeout 5000ms
      -   verifying given screenshot expectation
      - taking page screenshot
      -   disabled all CSS animations
      - waiting for fonts to load...
      - fonts loaded
      - Timeout 5000ms exceeded.

      89 |
      90 |   await expect(page.getByText("Wallet created successfully!")).toBeVisible();
    > 91 |   await expect(page).toHaveScreenshot("wallet-import-complete-page.png");
         |                      ^
      92 | });
      93 |
      94 | test("Import 24 word wallet", async ({ page }) => {

        at /workspaces/playwright-browsers/freighter/extension/e2e-tests/onboarding.test.ts:91:22

    attachment #1: wallet-import-complete-page-expected.png (image/png) ────────────────────────────
    e2e-tests/onboarding.test.ts-snapshots/wallet-import-complete-page-chromium-linux.png
    ────────────────────────────────────────────────────────────────────────────────────────────────

  1 flaky
    [chromium] › onboarding.test.ts:59:5 › Import 12 word wallet ───────────────────────────────────
  2 passed (24.4s)
Done in 25.32s.

increasing toHaveScreenshot's timeout to 10s fixed Import 12 word wallet.

We should not be retrying the error when expected image is missing, neither we should generate one unless --update-snapshots option is passed. We can fix that, this will make your tests fail earlier when the expectations are missing.

I see that when you provided linux expectations in the pr by renaming -darwin to -linux, the tests were failing with screenshots mismatch (see screenshot below), which is expected. Try generating the screenshots on linux, the error should disappear. If not, can you share a link to GHA output?

image

Also note that by using the same screenshot name "welcome-page.png" in multiple tests in the same file, you make all of them share the same expectation. Not sure if it's intentional.

yury-s commented 7 months ago

Further debugging revealed that Page.captureScreenshot CDP command never returns when capturing screenshot of a page with the extension. This is an issue in Chromium implementation of the command.

aristidesstaffieri commented 7 months ago

Further debugging revealed that Page.captureScreenshot CDP command never returns when capturing screenshot of a page with the extension. This is an issue in Chromium implementation of the command.

Thanks for digging into and the additional info. Do you happen to know if this is already an open issue on Chromium? Otherwise I can gather the details and open one with them.