[Bug]: Error: ENOENT: no such file or directory issue with Playwright upgrade from v1.24 to v1.45 in CI env

[ibrocodes7]

Version

1.45.0

Steps to reproduce

Background: We have an inbuilt test harness where all test files are run in parallel, based on the number of browser hosts provided. Currently, we use the Playwright v1.24 Docker image to create the browser environment, and everything functions smoothly with this setup.

After attempting the upgrade to Playwright v1.45, we are seeing numerous tests fail with the following error:

Error: ENOENT: no such file or directory, open '/home/admin/example/blink-playwright/test-results/.playwright-artifacts-0/traces/10fe2ef93047cfce1778-59ed9a256f901c93e72d-recording1.trace'.

These errors only occur in the CI environment that uses the Docker image and when the trace is set to "on" or "retain-on-failure." This issue is causing the tests to fail with the above error and not producing any logs, making it very difficult to debug.

As our test harness involves many internal systems, I won't be able to share the entire repository. However, I am willing to provide as many files and details as needed to help diagnose the issue. If there is any way I could debug this further, please let me know.

Your assistance in understanding why these errors are occurring and how we can resolve them would be greatly appreciated as we are completely blocked here.

Steps to reproduce:

1. Create browser image, necessary files found at https://github.com/ibrocodes7/playwright-bug.
2. Run the tests using npx playwright test --grep " "

Expected behavior

I expected the test to run smoothly, either failing or succeeding, and to generate the test results without causing the tests to break.

Actual behavior

I have inspected the contents of the test results folder and confirmed that the folder the test was looking for does not exist. I am attaching the complete text file of the test execution for your reference. The test ultimately failed due to a timeout.

Error file: https://github.com/ibrocodes7/playwright-bug/blob/main/pw-github/error-file.txt

Additional context

I’ve uploaded all these files to this GitHub repository - https://github.com/ibrocodes7/playwright-bug. included are the wrapper file responsible for orchestrating test execution in our CI environment, which sets up environment variables, processes command-line arguments, and initiates Playwright tests. Additionally, I've added 'build-browser-image' Groovy script, Playwright configuration, global setup, fixtures, and 'build.sh' files. Let me know if you need anything else.

Environment

  System:
    OS: Linux nebula-runs-fe-full-mode-236-browsers-by-jenkins-54b45c67c7vk8h 5.15.0-200.131.27.el8uek.x86_64 #2 SMP Wed Oct 4 22:19:10 PDT 2023 x86_64 x86_64 x86_64 GNU/Linux
  CPU:
  Model name:                         Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz
  Memory:
  Mem:           6.0Gi        15Mi       6.0Gi        12Gi       1.0Mi       6.0Gi

  Binaries:
    Node: v18.20.2
    npm: 9.5.1
  Languages:
    GNU bash, version 5.1.16(1)-release (x86_64-pc-linux-gnu)
  npmPackages:
    @playwright/test: 1.45.0 => 1.45.0 

Running npx envinfo --preset playwright does not produce any output in the CI environment, so I have manually gathered and entered the details using individual commands.

[dgozman]

@ibrocodes7 I cannot access https://github.com/ibrocodes7/playwright-bug, could you please double check it?

[ibrocodes7]

@dgozman please check now.

[mxschmitt]

I think we fixed a similar error in 1.45.1. Please check in v1.45.1 and Canary.

[ibrocodes7]

Hi @mxschmitt @dgozman I am still encountering the error. I have tried with all the images listed below: mcr.microsoft.com/playwright:v1.45.1 mcr.microsoft.com/playwright:v1.45.1-jammy mcr.microsoft.com/playwright:v1.45.1-focal mcr.microsoft.com/playwright:next-jammy mcr.microsoft.com/playwright:next-focal mcr.microsoft.com/playwright:next

Error details: https://github.com/ibrocodes7/playwright-bug/blob/main/pw-github/1.45.1-error.txt

Could you please take a look?

[dgozman]

@ibrocodes7 I looked over the repository you provided. There are no test files there, and files like global-setup.ts import from the src directory which is also missing. I guess we'll also need at least the BASE_URL to be able to run the tests.

Basically, without being able to reproduce the issue, we won't be able to debug it. Please make sure to share everything needed and clear instructions on how to reproduce.

[ibrocodes7]

Hi @dgozman

I have created a repository with demo tests using Playwright's example tests, which utilize the Playwright website as the URL. This reproduced the same error in the Jenkins Docker environment.

Please review the error file and the repository here: https://github.com/ibrocodes7/playwright-upgrade-issue-repo/.

Kindly run the tests in the specified environment details and with multiple browser hosts in Jenkins as mentioned above. Thank you

[dgozman]

@ibrocodes7 Thank you for the better repro! However, I am still not sure how to reproduce. I don't have a Jenkins account nor do I have any repositories with CI runs on Jenkins. I don't think I'll be able to reproduce the issue on Jenkins. Do you have a repro that I can run locally on my computer?

Aside from the repro, perhaps you are running out of disk space during the tests? That could explain the errors with accessing the filesystem.

[pavelfeldman]

We need more information to act on this report. As long as we can't repro it, it is unlikely with can make progress with it. Allow me to close it, but please file a new one and link to this issue when you get back to it!