search

webdriverio-community / wdio-vscode-service

A service to test VSCode extensions from end to end using WebdriverIO
https://webdriverio-community.github.io/wdio-vscode-service/
MIT License
37 stars 28 forks source link

E2E tests fail in VSCode version 1.93.0 with "target window already closed" when running "Developer: Reload Window" #136

Open daphne-sfdc opened 2 months ago

daphne-sfdc commented 2 months ago

Hi WDIO Team,

I'm working on the Salesforce Extensions for VSCode. Our E2E tests are built using the WDIO framework. After the release of VSCode version 1.93.0, my team started to notice that our E2E tests are failing with a "target window already closed" error. We are using the latest version v6.1.0 of wdio-vscode-service.

Here's an example of a failed test run on 1.93.0: https://github.com/forcedotcom/salesforcedx-vscode/actions/runs/10779098563/job/29891858831

Currently, we have pinned our E2E tests to run on VSCode version 1.92.2, which is a temporary workaround. However, we do want to keep up with the times and run E2E tests on stable VSCode version.

I was able to reproduce the issue when running my E2E test locally. I found that the issue happens when running Developer: Reload Window in the target VSCode window. As soon as I hit Enter after selecting Developer: Reload Window from the command palette, the target VSCode window gets closed. I tried several other commands, and they ran fine in the target window.

Here's a short video of the issue: https://vimeo.com/1008588498

seanpoulter commented 2 months ago

Thanks for reporting the issue @daphne-sfdc. To get ahead of @christian-bromann, any contributions would be much appreciated!

I'm not familiar with the specifics, but I would expect the Developer: Reload Window command to behave differently from other commands since it will close the VS Code / Electron / Browser window that it is connected to so we'd no longer be able to communicate with the target window. You'd have to detect the new window and update the reference in all the selectors. With some luck, other maintainers have already encountered this problem here or in the Electron service too.

daphne-sfdc commented 2 months ago

Hi @seanpoulter,

Thank you for getting back to us. My team and I are looking into the issue further and were wondering if you could provide any additional details about what changed in VSCode version 1.93.0. Specifically, we are interested in understanding what might be causing the selectors to no longer detect the new target window after a reload.

We have already reviewed the VSCode release notes but didn't find any information regarding changes to selectors. Any insights or pointers you could share would be greatly appreciated, as they would help us in addressing and potentially fixing the bug.

Thank you!

haberdashPI commented 2 months ago

FYI: a workaround is to install wdio tools at version 8 and wdio-vscode-service at version 6.0.3. If either was a later version I got the same error, but with those version everything ran fine.

daphne-sfdc commented 2 months ago

Hi @haberdashPI, Thank you for suggesting the workaround. Unfortunately, it didn't resolve the issue for me. I was already using all WDIO tools at version 8, and downgrading wdio-vscode-service to v6.0.3 still resulted in the "target window already closed" error after running the Developer: Reload Window command.

haberdashPI commented 2 months ago

You might want to check through the errors above that, that is usually the product of some problem further upstream, so that might have more clues

haberdashPI commented 2 months ago

Out of curiosity I looked through your tests: I don't think it has anything to do with my problems.

[0-0] Reloading window

After that message, the test never recovers; seems like the root issue is about how/if the service re-attaches when reloading the app.

daphne-sfdc commented 2 months ago

Hello @seanpoulter,

I followed your suggestion to check the selectors but found during my investigation that the selector for the entire VSCode window (.monaco-workbench) did not change between VSCode versions 1.92.2 and 1.93.0.

I then probed deeper into the code to check how wdio-vscode-service handles Developer: Reload Window and was quite surprised to find that reload doesn't seem to be handled at all in the code. I also put console.log() statements all over the code that handles opening VSCode at the beginning of the E2E test (in src/service.ts and src/launcher.ts), figuring a reload would be handled the same way as the first time VSCode is opened, but none of my logs got printed during a successful reload that I ran with 1.92.2.

This leads us to believe that the issue is not with the selectors but rather a larger design issue in wdio-vscode-service. It appears that reloads were never handled in the first place.

We would appreciate your guidance on how to proceed with this. It seems like a significant change might be required to properly handle window reloads in wdio-vscode-service.

Thank you for your assistance.

daphne-sfdc commented 2 months ago

Hi @haberdashPI, Do you have examples of tests containing Developer: Reload Window that you can share with us? We'd like to take a look to see how that works in VSCode version 1.93. Thanks!

haberdashPI commented 2 months ago

I do not. I have not tried to use this package under that scenario.

CristiCanizales commented 2 months ago

Hi @seanpoulter, I was wondering if you had any comments about what @daphne-sfdc mentioned above. Thanks again in advance for your help.