search

microsoft / WinAppDriver

Windows Application Driver
MIT License
3.69k stars 1.4k forks source link

Problems using WAD with Azure Pipelines #1380

Open Jteve-Sobs opened 3 years ago

Jteve-Sobs commented 3 years ago

Hi,

Does anyone else use WAD in an Azure DevOps pipeline to execute tests and is having problem with flaky tests? I noticed that tests regularly fail at my side but when executing them locally on the same testing environment everything works fine. At this point I don't exactly know what can be further improved in the tests. Any advice working with WAD in addition with the pipline feature of Azure would be great.

I understand this is not an issue with WAD but I didn't know where to ask elsewhere. If this does not belong here then close this issue.

anunay1 commented 3 years ago

Even I am facing the same issue, sendkeys start behaving weird, it will start send weird characters. Also I see stability is decreased. What are you using are you sing agent jib task or starting WinAppdriver.exe programmatically.

Jteve-Sobs commented 3 years ago

I currently don't know if the WebDriver SendKeys method is working because I use the WindowsForms SendKeys.SendWait method. I observed this behavior before and changed the way to do inputs. At the moment I am mostly struggling with elements that are not found but are definitly there (also verified by getting screenshots when an error occurs) I don't have a task setup because I check at the start of my testrun that WinAppDriver.exe is running and else starting it. Found this is the best method to set it up.
How do you using to startup WinAppDriver.exe ?

anunay1 commented 3 years ago

Can you check what is the text size of the system on which the tests are running it should be 125%

Jteve-Sobs commented 3 years ago

Are you sure that the scaling should be 125% ? This would be rather inconvinient for me because one application that is being tested should be running with 100% scaling. In addition some testing environments can only the accessed via RDP so changing that setting is tied to the host OS.

anunay1 commented 3 years ago

Can you check what is the text size in the PC where you are running the tests with azure.

Jteve-Sobs commented 3 years ago

So you mean I should set the first slider to 125% ? image

anunay1 commented 3 years ago

Yes

Jteve-Sobs commented 3 years ago

Ok, will try this. Thanks.

VijayHugar commented 3 years ago

@Jteve-Sobs Since you are executing through Azure DevOps, make sure you are running agent in interactive mode and also, have a azure task for setting up screen resolution as shown below. This is how we had setup and it is working on daily basis.

image

Jteve-Sobs commented 3 years ago

Thanks for your advice. The agent is not running as a service so I guess it is in interactive mode (I have not installed it by myself). How can I check if the agent is installed in interactive mode? The task "Set Screen Resolution" is added to the pipeline. However with some virtual machines I get this error:

image

I will have to see how this can be fixed.

anunay1 commented 3 years ago

If you see the agent running as a command line prompt then it's not running like a service.

anunay1 commented 3 years ago

Thanks for your advice. The agent is not running as a service so I guess it is in interactive mode (I have not installed it by myself). How can I check if the agent is installed in interactive mode? The task "Set Screen Resolution" is added to the pipeline. However with some virtual machines I get this error:

image

I will have to see how this can be fixed.

You need to set the resolution that is supported as given in the list, did the set text to 125 % work

Shakevg commented 3 years ago

@Jteve-Sobs To avoid problems on VM with Azure Agent I usually trying to find controls using Inspector.exe, debug and check tests on the Agent machine because markup can be not the same if the machine is different (e.g. different build of Windows OS). Besides, I enable video recording for failed tests to investigate such issues.

Also, could you describe in more detail what are flaky issues did you face?

Shakevg commented 3 years ago

Thanks for your advice. The agent is not running as a service so I guess it is in interactive mode (I have not installed it by myself). How can I check if the agent is installed in interactive mode? The task "Set Screen Resolution" is added to the pipeline. However with some virtual machines I get this error:

image

I will have to see how this can be fixed.

I saw such an error in 2 casses:

  1. You set the resolution that is not supported as given in the list (if you need a higher resolution than in the list, then it should be increased in VM settings).
  2. During task execution you connected using RDC with lower resolution than trying to set.

Also if you will select the "Use optimal resolution" option it will not fail, but it will be a max available resolution.

Jteve-Sobs commented 3 years ago

If you see the agent running as a command line prompt then it's not running like a service.

Ok, I got this. Thanks.

You need to set the resolution that is supported as given in the list, did the set text to 125 % work The best would be to set it to 1920 by 1080. Unfortunately this is not possible at the moment.

It did not help that much. I ran a few testruns and it the same as before from what I was seeing

I will try the video recording but I already take a screenshot when a test fails and have enough messages to track the exact point where it fails.

My main problem is still that a local test execution via PowerShell is succeeding most of the time and the execution via pipeline fails more often. Will have to investigate further. Thanks all for the advices.