search

webdriverio-community / wdio-electron-service

WebdriverIO service for testing Electron applications
https://webdriver.io
MIT License
32 stars 15 forks source link

Tests fail in a project with splashscreen #733

Open RedX2501 opened 3 weeks ago

RedX2501 commented 3 weeks ago

I have a project that runs fine with Webdriver v8.40.3 and wdie-electron-service 6.

I'm trying to migrate to v9.

I have tried using 7.0.0-beta.10 but that did not solve the problem.

I have created a fresh wdio test project to make sure it's not a cnofiguration error.

I'm using electron 29.1.4.

describe('Electron Testing', () => {
    it('should print application title', async () => {
        await browser.pause(10_000); // Added so that the main window has time to load.
        console.log('Hello', await browser.getTitle(), 'application!')
    })
})

Error:

[0-0] 2024-08-20T06:51:03.421Z INFO webdriver: COMMAND getWindowHandles()
[0-0] 2024-08-20T06:51:03.421Z INFO webdriver: [GET] http://localhost:55479/session/d39baa3e59c3e5c9c8035d7328329e58/window/handles
[0-0] 2024-08-20T06:51:03.424Z INFO webdriver: RESULT [ '3F1FAD1B8DFC8444955D7546C4B2D76B' ]
[0-0] 2024-08-20T06:51:03.425Z INFO HtmlReporter: onRunnerStart: 0-0
[0-0] 2024-08-20T06:51:03.427Z INFO HtmlReporter: onHookStart: 0-0:hook-0-0
[0-0] 2024-08-20T06:51:03.427Z INFO HtmlReporter: onHookEnd: 0-0:hook-0-0
[0-0] 2024-08-20T06:51:03.428Z INFO HtmlReporter: onSuiteStart: 0-0:suite-0-0
[0-0] 2024-08-20T06:51:03.428Z INFO HtmlReporter: onTestStart: 0-0:test-00-0
[0-0] 2024-08-20T06:51:13.437Z INFO webdriver: COMMAND getTitle()
[0-0] 2024-08-20T06:51:13.437Z INFO webdriver: [GET] http://localhost:55479/session/d39baa3e59c3e5c9c8035d7328329e58/title
[0-0] 2024-08-20T06:51:13.442Z WARN webdriver: Request failed with status 404 due to no such window: target window already closed       
[0-0] from unknown error: web view not found
[0-0]   (Session info: chrome=122.0.6261.129)
[0-0] 2024-08-20T06:51:13.442Z INFO webdriver: Retrying 1/3
[0-0] 2024-08-20T06:51:13.442Z INFO webdriver: [GET] http://localhost:55479/session/d39baa3e59c3e5c9c8035d7328329e58/title
[0-0] 2024-08-20T06:51:13.444Z WARN webdriver: Request failed with status 404 due to no such window: target window already closed       
[0-0] from unknown error: web view not found
[0-0]   (Session info: chrome=122.0.6261.129)
[0-0] 2024-08-20T06:51:13.444Z INFO webdriver: Retrying 2/3
[0-0] 2024-08-20T06:51:13.444Z INFO webdriver: [GET] http://localhost:55479/session/d39baa3e59c3e5c9c8035d7328329e58/title
[0-0] 2024-08-20T06:51:13.446Z WARN webdriver: Request failed with status 404 due to no such window: target window already closed       
[0-0] from unknown error: web view not found
[0-0]   (Session info: chrome=122.0.6261.129)
[0-0] 2024-08-20T06:51:13.446Z INFO webdriver: Retrying 3/3
[0-0] 2024-08-20T06:51:13.446Z INFO webdriver: [GET] http://localhost:55479/session/d39baa3e59c3e5c9c8035d7328329e58/title
[0-0] 2024-08-20T06:51:13.447Z ERROR webdriver: Request failed with status 404 due to no such window: no such window: target window already closed
[0-0] from unknown error: web view not found
[0-0]   (Session info: chrome=122.0.6261.129)
[0-0] no such window in "Electron Testing.should print application title"
no such window: no such window: target window already closed
from unknown error: web view not found
  (Session info: chrome=122.0.6261.129)
    at Context.<anonymous> (c:\data\_eev\testproj\test\specs\test.e2e.ts:6:30)
[0-0] 2024-08-20T06:51:13.448Z INFO HtmlReporter: onTestFail: 0-0:test-00-0
[0-0] 2024-08-20T06:51:13.449Z INFO HtmlReporter: onTestEnd: 0-0:test-00-0
[0-0] 2024-08-20T06:51:13.449Z INFO HtmlReporter: onSuiteEnd: 0-0:suite-0-0
[0-0] 2024-08-20T06:51:13.449Z INFO HtmlReporter: onHookStart: 0-0:hook-0-1
[0-0] 2024-08-20T06:51:13.449Z INFO HtmlReporter: onHookEnd: 0-0:hook-0-1
[0-0] 2024-08-20T06:51:13.449Z INFO webdriver: COMMAND deleteSession()
[0-0] 2024-08-20T06:51:13.450Z INFO webdriver: [DELETE] http://localhost:55479/session/d39baa3e59c3e5c9c8035d7328329e58
[0-0] 2024-08-20T06:51:13.486Z INFO webdriver: RESULT null
[0-0] 2024-08-20T06:51:13.486Z INFO webdriver: Kill driver process with PID 36272
[0-0] 2024-08-20T06:51:13.488Z INFO HtmlReporter: onRunnerEnd: 0-0
[0-0] 2024-08-20T06:51:13.488Z INFO HtmlReporter: onRunnerEnd: 0-0 awaiting report generation
[0-0] 2024-08-20T06:51:13.488Z INFO ReportGenerator: Report Generation started
[0-0] 2024-08-20T06:51:13.488Z INFO ReportGenerator: Included metrics for suite: 0-0 suite-0-0
[0-0] 2024-08-20T06:51:13.489Z INFO ReportGenerator: Generated 1 specs, 1 suites,
[0-0] 2024-08-20T06:51:13.489Z INFO JsonGenerator: Json stringify starting: C:\data\_eev\testproj\reports\html-reports\report-0-0.json  
[0-0] 2024-08-20T06:51:13.501Z INFO JsonGenerator: Json stringify complete
[0-0] 2024-08-20T06:51:13.502Z INFO JsonGenerator: Json write completed: C:\data\_eev\testproj\reports\html-reports\report-0-0.json     
[0-0] 2024-08-20T06:51:13.503Z INFO HtmlGenerator: Html Generation started
[0-0] 2024-08-20T06:51:13.503Z INFO HtmlGenerator: Html Generation started in C:\data\_eev\testproj\node_modules\wdio-html-nice-reporter[0-0] 2024-08-20T06:51:13.516Z INFO HtmlGenerator: Html Generation Completed
[0-0] 2024-08-20T06:51:13.517Z INFO ReportGenerator: Report Generation completed
[0-0] 2024-08-20T06:51:13.593Z INFO HtmlReporter: isSynchronized: true
[0-0] FAILED in chrome - file:///C:/data/_eev/testproj/test/specs/test.e2e.ts
2024-08-20T06:51:13.606Z INFO @wdio/cli:launcher: Run onWorkerEnd hook
2024-08-20T06:51:13.606Z INFO @wdio/cli:launcher: Run onComplete hook

 "spec" Reporter:
------------------------------------------------------------------
[chrome 122.0.6261.129 windows #0-0] Running: chrome (v122.0.6261.129) on windows
[chrome 122.0.6261.129 windows #0-0] Session ID: d39baa3e59c3e5c9c8035d7328329e58
[chrome 122.0.6261.129 windows #0-0]
[chrome 122.0.6261.129 windows #0-0] » \test\specs\test.e2e.ts
[chrome 122.0.6261.129 windows #0-0] Electron Testing
[chrome 122.0.6261.129 windows #0-0]    ✖ should print application title
[chrome 122.0.6261.129 windows #0-0]
[chrome 122.0.6261.129 windows #0-0] 1 failing (10s)
[chrome 122.0.6261.129 windows #0-0]
[chrome 122.0.6261.129 windows #0-0] 1) Electron Testing should print application title
[chrome 122.0.6261.129 windows #0-0] no such window: target window already closed
from unknown error: web view not found
  (Session info: chrome=122.0.6261.129)
[chrome 122.0.6261.129 windows #0-0] no such window: no such window: target window already closed
[chrome 122.0.6261.129 windows #0-0] from unknown error: web view not found
[chrome 122.0.6261.129 windows #0-0]   (Session info: chrome=122.0.6261.129)
[chrome 122.0.6261.129 windows #0-0]     at Context.<anonymous> (c:\data\_eev\testproj\test\specs\test.e2e.ts:6:30)

Spec Files:      0 passed, 1 failed, 1 total (100% completed) in 00:00:14

The project uses a splash screen that closes once the main window successfully connects to the backend.

goosewobbler commented 3 weeks ago

There's been some issues getting the next major out, see #533 for details. The betas have been tested with WDIO v9 in wdio-electron-service-example and are working there.

It sounds like Chromedriver is trying to connect to the splash screen. I'm not sure why this would change with WDIO v9...

goosewobbler commented 3 weeks ago

v7 of the service has been released now, this only works with WDIO v9. Can you try it again with the latest builds? Presumably the tests work when the splash screen is disabled?

RedX2501 commented 3 weeks ago

How does chromedriver know whether somthing is a splash screen or de main window?

goosewobbler commented 3 weeks ago

Do you have a repo to look at? It's not easy to debug without seeing what the app is actually doing.

RedX2501 commented 3 weeks ago

I have now :) See https://github.com/RedX2501/electron-react-test/commit/c30f1f3e621a468e5021d2bcb6e27f41b50de672

I have a separate project in which the tests are. I will package the electron app and then use appBinaryPath to test. This setup is necessary as the tool is developed by a third-party and we receive the source code and binaries on release.

I write acceptance tests for the binaries we receive to make sure they comply with our specs.

RedX2501 commented 3 weeks ago

I have tried with the released v7 and it's still the same problem.

If I change the code to find all the handles and then switch to the only handle available it actually works.

This behaviour I know from the v6 version, although I think it's counter intuitive.

I need to enumerate all handles because calling browser.getHandle() actually throws when the first window (splash) has closed and only the second windows is showing.

The error is

no such window: target window already closed
from unknown error: web view not found
  (Session info: chrome=122.0.6261.129)
goosewobbler commented 3 weeks ago

So, does it work if you disable the splash screen?

Also, you mention the approach of changing the code to switch handles was known from the previous version. Does this mean the error also occurs in WDIO v8 if you do not do this step?

I'm just trying to properly define this issue as it is clear the splash screen use case should be supported, but it's not obvious how this relates to the new v9 release (if it does at all).

I'm going to refer only to WDIO versions going forward, you should be using v7.x of the service with v9.x of WDIO, and v6.x of the service with v8.x of WDIO.

RedX2501 commented 2 weeks ago

Hi,

I have tested it some more.

If I have only one window it does work. So having only the Splashscreen and never opening MainWindow works fine. Having only MainWindow and never opening the Splashscreen also works. Having Splashscreen stay open long enough for the test to run to completion also works.

So basically it boils down to my expectation that browser.getTitle() should just work if there is one window only.

I'm fine with closing this ticket as wrong expectation.

Should I open another one about the getTitle (and other functions) failing when the first window closes? In this repo or another? Is this behaviour documented somewhere?

goosewobbler commented 2 weeks ago

So it definitely sounds like the display time of the splash is too short for WDIO to run tests on the splash, which results in the target window closed error.

I would have a separate splashscreen.spec.ts file where you run tests which assert properties of the splash, you might also assert that it disappears after a certain time, for this spec you will likely need to operate your splash in a "test mode" where you can customise the display time, how long the splash is displayed before it disappears.

Then all other spec files could either wait until the splash is replaced with the main window, or simply run the app with a splash display time of 0 - so the splash is replaced with the main window immediately - before running their tests.

You could search (or ask) in the main WDIO forum about how best to test projects with splash screens, I'm sure there will have been others with the same issue. There is also the WDIO Discord where you can ask questions around testing with WDIO.

We could consider some kind of splash screen helper in the service but I'm not sure it would save much time or code, the solution here seems to be an improved structure for your tests to test the splash separately from the main window.

RedX2501 commented 2 weeks ago

I think i didn't express myself correctly.

I don't want to test the splashscreen, what's undocumented/weird ist that even if i wait long enough for the main window to appear browser.getTitle fails.

Only after switchToWindow it works again.

I would have expected to not need to explicitly call switchToWindow when getHandles returns only one entry.

goosewobbler commented 2 weeks ago

Ok, I see. That sounds like something which could have a solution attached. So you're saying that you need to explicitly call switchToWindow after the splash is closed, in order to switch to the main window from the splash.

@christian-bromann Do you think there's a feature request here? It would make sense if WDIO could auto-switch windows in this splash screen case.

RedX2501 commented 2 weeks ago

This affects many other functions too even getHandle (without s). It won't retrieve the current handle of the single window if one window closed and another opened. It only starts to work after switchToWindow.

goosewobbler commented 2 weeks ago

Possibly related: https://github.com/webdriverio/webdriverio/issues/13436

christian-bromann commented 2 weeks ago

Do you think there's a feature request here? It would make sense if WDIO could auto-switch windows in this splash screen case.

I would assume that the WebDriver protocol automatically switches the window handle if a window closes. Let me verify that.