microsoft / playwright

Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.
https://playwright.dev
Apache License 2.0
67.34k stars 3.71k forks source link

[Bug]: ElectronApplication.evaluate() is unreliable #33737

Open jjeff opened 1 week ago

jjeff commented 1 week ago

Version

1.49.0

Steps to reproduce

  1. Create an Electron Application
  2. Package it
  3. Launch it with Playwright
  4. Run a test similar to the following:
test('stress test: run electronApp.evaluate() over and over', async () => {
  const app = getApp()
  const iterations = 1000
  for (let i = 0; i < iterations; i++) {
    const displays = await app.evaluate(({ screen }) => screen.getAllDisplays())
    expect(displays).toBeTruthy()
    expect(displays.length).toBeGreaterThan(0)
  }
})

Expected behavior

No errors

Actual behavior

When communicating with main or renderer processes, I regularly get errors similar to the following:

These errors seem to happen randomly. And they will also happen with methods like electronApplication.getBrowser(page) or page.evaluate()

Additional context

This seems to have started with the release of Electron 27 and has continued in subsequent versions.

I know that Electron support in Playwright is experimental. And I've been writing some workarounds for the electron-playwright-helpers library. But before I release the workarounds, I'd like to figure out what's happening.

I have yet to create a minimal reproduction that generates these errors nearly as reliably as my (maximal) application.

Environment

System:
    OS: macOS 15.1.1
    CPU: (12) arm64 Apple M2 Max
    Memory: 262.55 MB / 32.00 GB
  Binaries:
    Node: 20.15.0 - ~/.nvm/versions/node/v20.15.0/bin/node
    Yarn: 1.22.18 - /usr/local/bin/yarn
    npm: 7.24.2 - ~/Code/missioncontrol/node_modules/.bin/npm
  IDEs:
    VSCode: 1.95.3 - /usr/local/bin/code
  Languages:
    Bash: 3.2.57 - /bin/bash
  npmPackages:
    @playwright/test: 1.49.0 => 1.49.0 
    playwright: 1.49.0 => 1.49.0
mxschmitt commented 1 week ago

We would like to help you - would it be possible to share a minimal repro?