search

appium / dotnet-client

Extension to the official Selenium dotnet webdriver
Apache License 2.0
383 stars 188 forks source link

[Bug]: WindowsDriver throws NRE on random Test execution on random UI Tests. #569

Closed Stadzior closed 1 year ago

Stadzior commented 1 year ago

Description

Prologue: Hi, I'm banging my head with this issue for few days now. I'll give you as much context as I can and I'm really looking forward to get any hints/clues/tips that can at least push me in the right direction. Feel free to ask for further details if that might help. AFAIK the issue is not related to the code of the tests but rather to configurations and setup, but I'm not sure about anything right now. I know that stating such vague bug report without proper repro steps is not welcomed, nonetheless I need to seek for assistance somewhere and I feel that this issue fits here best.

Issue description: We are using Azure DevOps pipelines & Azure VM to run our UI Tests for .NET6 WPF apps. I have two Apps that are "UI tested" with the use of Appium.WebDriver. One of them - let's call it "The Good App" has no issues with the UI tests. They are running perfectly fine. The other one - let's call it "The Bad App" crashes during test executions. And that's all what I can say for sure. It happens randomly without any pattern on some test executions and on some UI Tests.

Environment

OS on VM: Windows Server 2019 Datacenter version: 10.0.17763 build: 17763 arch: x64 OS on my PC: Windows 10 Pro version: 21H2 build: 19044.2251 arch: x64 Appium.WebDriver version: 4.4.0 NUnit version: 3.13.3 NUnit TestAdapter version: 4.3.1 vstest version: 17.4.1 VSTest@2 task version: 2.210.0 Apps .NET version: 6.0 Visual Studio version: 17.4.0 Enterprise 2022 x64

Details

I have tried the following:

Nothing. No pattern revealed, no questions answered. You can check stacktrace and example pipeline step logs in the attached gist. It all fails on creation of WindowsDriver in DesktopSession.cs.

You might want to say - maybe it's an Azure DevOps pipeline issue and you should just create new clean pipeline and try to run the tests there? Well I thought about that but it's not related to the Azure DevOps itself because I am able to reproduce this issue on my PC but only if I DEBUG tests in Visual Studio Test Explorer. If I select the RUN option it works just fine EVERYTIME.

One UI Test that is checking if the app has opened and displays username is present in both apps and in "The Good App" it works and in "The Bad App" it also fails randomly. There is no difference between versions in nugets nor dotnet in both of those apps. I have also added the anonymized code of the this very UI test to the gist.

Code To Reproduce Issue [ Good To Have ]

DasboardTests.cs & DesktopSession.cs & AppDesktopSession.cs in: https://gist.github.com/Stadzior/daec8b22307f505b1829730f7302e929

Exception stack traces

stack_trace.txt in: https://gist.github.com/Stadzior/daec8b22307f505b1829730f7302e929

Link to Appium logs

Not sure if those are the logs that you're asking for but I've attached pipeline step logs in pipeline_step_logs.txt file in: https://gist.github.com/Stadzior/daec8b22307f505b1829730f7302e929

Dor-bl commented 1 year ago

@Stadzior, The error I saw from your logs is : OpenQA.Selenium.WebDriverException : Object reference not set to an instance of an object.

  1. Did you made sure the windows driver is not losing focus/ connection?
  2. Did you tried updating to the beta version of Appium.WebDriver? Although I doubt it's related to the appium .NET client
  3. Are you aware that the microsoft windows driver is not supported for a while now? Is strongly advise you try to use the Appium WinApp driver instead.
Stadzior commented 1 year ago

Hi @Dor-bl.

  1. I have e.g. monitored the WinAppDriver session under e.g. http://127.0.0.1:4723/session/SESSION-ID and it works fine as long as the OpenQA.Selenium.Appium.Windows.WindowsDriver object is created. But as you can see in the stacktrace - the problem occurs in the ctor of this type. I am pretty sure that it is not a losing focus problem too because "The Good App" is working just fine and also "The Bad App" is working fine when I use "Run Unit Tests" option instead of "Debug Unit Tests". Can you please give me your thoughts on how to make sure that the windows driver is not losing focus/ connection any other way?

  2. Yes, and It threw a lot compilation errors on me (probably some breaking changes?)

  3. We are not using it. Can you point me to an article on how to migrate? I can't really find one. I am using an environment variable when running WinAppDriver.exe is there an equivalent for it when running appium command from cli?

Stadzior commented 1 year ago

@Update unfortunetelly after the migration the problem remains (I guess WindowsDriver exe is running under the hood). OpenQA.Selenium.WebDriverException: 'An unknown server-side error occurred while processing the command. Original error: Object reference not set to an instance of an object.'

It really seems to be a very vague problem with windows driver server.

Stadzior commented 1 year ago

I'll try to forward it to https://github.com/microsoft/WinAppDriver/issues

Dor-bl commented 1 year ago

Closed as a third-Party issue (WinAppDriver)