[Feature] Headless Electron

[bpasero]

I could not find a way to run Electron tests headless like that is possible with browser tests.

//cc @pavelfeldman

[pavelfeldman]

Electron itself does not have the headless mode. It can probably be simulated via hiding the window, but there is no built-in functionality of swapping the graphics stack like there is in Chromium. What do your existing tests do today?

[bpasero]

What do your existing tests do today?

:-), got me there. Yeah, not headless. I think its fine, I was just thinking that browser and electron should support the same set of options, but its not a big deal for us I would say.

In our CI though we do run headless browser tests and maybe that means a bit faster execution time, but I see the issue that Electron itself does not really support this mode.

[pavelfeldman]

It wasn't a gotcha reply. I was hoping you have an ad-hoc solution, something like hiding the window in test mode, that was working for you.

[tanishqkancharla]

I was able to get headless electron tests working with this snippet: override the default behavior of showing the browser window once it's ready to show:

this.browserWindow.once("ready-to-show", () => {
    if (config.test?.headless) return
    this.browserWindow.show()
})

So you can run the app electron main.js --test-headless=true, parse the process.argv, and when actually ready to display a window, you read the config file, and defer showing the window. This also means you can also test "head-fully" if, for example, playwright is invoked in DEBUG mode:

const isDebug = Boolean(process.env.PWDEBUG)

const config: Partial<Config> = {
  test: {
      headless: !isDebug,
  },
  partition,
}

const args = [rootPath("build/main.js"), ...makeAppCliArgs(config)]

The renderer process is still running, just completely headlessly, without a display, so the tests are all valid. For this to work on CI (on linux), you need xvfb.

Also these would be great comments to go on the docs.

[greenwood-vaarst]

@tanishqkancharla can I ask are you then still able to do

const app = await electron.launch(someConfig);
const page = await app.firstWindow();

or are you creating a different kind of page somehow?

thanks

[tanishqkancharla]

@greenwood-vaarst that's exactly what I'm doing. I use fixtures, and the api comes out pretty clean. Here's a snippet of what that looks like:

export const e2eTest = baseTest.extend<E2ETestFixture, E2EWorkerFixture>({
    testData: [
        async ({ harness }, use, testInfo) => {
                        // ...
            // Run head-full if debugging
            // npx playwright --debug
            const isDebug = Boolean(process.env.PWDEBUG)

            const config: Partial<Config> = {
                test: {
                    MAIN_PORT,
                    RENDERER_PORT,
                    headless: !isDebug,
                    id: testId,
                },
                partition,
            }

            // ...

            const app = await electron.launch({ args, colorScheme: "dark" })
            await app.firstWindow()

            await harness.waitUntilTestIsReady(testId)

            try {
                await use({ app, harness, partition, testId })
            } finally {
                harness.dispatch.unmountTest(testId)
                await app.close()
            }
        },
        { auto: true, timeout: 5000 },
    ],
})

And here's how it gets used:

e2eTest("Finds match", async ({ testData }) => {
        const h = e2eDialect(testData)
        const w = h.window(0)

        await h.createHtmlFile("test.html", `<p>some words</p>`)
        await w.openFile("test.html")
                // ...
})

When I was working on this project, I didn't really use the playwright page object directly very much. Instead I hooked up a third test harness server that connected to both the main and renderer process and communicated with them directly. In the second code snippet, h.createHtmlFile calls a remote handler on the main process and w is an alias to the first window.

[greenwood-vaarst]

Thanks so much for your reply @tanishqkancharla