WinAppDriver cannot find elements (neither can inspect)

[DGCBio]

We have run into a strange problem where some elements cannot be found by WinAppDriver, yet they appear on the screen. In debugging the issue, we have determined that they are not properly being added to the UI tree in inspect either.

There is a listview in our app that is on the landing screen. In our automated tests, the listview is almost never able to be found by WinAppDriver, which then causes the tests to fail.

Opening the app manually and viewing it with inspect shows some 'custom' nodes that do not have the children that we expect (although we can clearly see them on the screen). After navigating away from this screen and returning, the elements are added as expected, but this is not an acceptable workflow to ensure automation works for our testing.

In the image below, the left side shows the tree upon first arrival of the listview. The list should be within the 'custom' node in red. On the right side shows the tree after returning. The list is there along with several other nodes including the one listview item.

However, when using inspect, even on first arrival, we CAN use the mouse to select the listview. When we do this, the item details are shown including our assigned automation id and the element information. Except it does not auto-navigate to anywhere on the treeview on the left side because evidently there is no parent in that tree. It seems to be an orphaned element.

Unfortunately, I cannot post screenshots of the application as it contains sensitive product information (and I have blocked out such text).

We aren't sure this is a WinAppDriver issue, since the problem is also perceived in inspect. But we don't know who else to bring this to. Any advice/help would be greatly appreciated.

Windows 10 1903 build 18362.329 WinAppDriver v 1.1.3.0

[DLightstone]

Dump the actual XML rather than relying in inspect.

Something like the code below inserted appropriately wilt give you the various xml tag names GuiRootAccess.FindWindowSession(window, 1); gets the session corresponding the title of the Window containing the control of interest

Once you have the xml you can determine the tags and search keys needed for searchs by XPATH to navigate item by item to the control of interest.

If you can't find the tag path you want, it probably is not yet implemented by Microsoft (and might never be implemented)

public static void SaveSourceToFile(string window, string strFile) { WindowsDriver session; bool bExists; session= GuiRootAccess.FindWindowSession(window, 1); Assert.IsNotNull(session); var source = session.PageSource; Assert.AreNotEqual(source.Length, 0); bExists = Utilities.OSFileUtilities.DoesFileExist(strFile); if (bExists) { Utilities.OSFileUtilities.DeleteFile(strFile); } System.Xml.XmlDocument xmlDoc = new System.Xml.XmlDocument(); System.Xml.XmlTextWriter w = new System.Xml.XmlTextWriter(strFile, System.Text.Encoding.BigEndianUnicode); xmlDoc.LoadXml(source); xmlDoc.WriteTo(w); w.Close(); }

[DGCBio]

@DLightstone,

Thanks for the reply.

We have checked the xml page source and we see the exact same issue. I didn't post that because it seemed redundant, but the page source also does not have the node in the tree.

[DLightstone]

New Text Document.txt My wild guess is that Microsoft has not yet implemented a means to correctly identify the control you wish to have identified.

If you play with the TreeView control at all you will quickly discover that

(1) image icons, checkboxes, and node tags are "invisible" (is no xml tags suitable for observation) (2) node expand/contract and checkboxes not considered to be a controls ( or subcontrol depending on perspective.)

Necxt there is a possibility that a different set of project build options my expose a different interpretation of the items (See #849)

Finally the missing information may be a attribute list With DataGrids each row is something like the contents of the file below

After deleting a lot of attributes focus in on the name attribute. its text value may be what you want

New Text Document.txt

With the tag DataItem corresponding to data table attribute I have associated the tag Custom with structure instance and DataItem with structure field instance

[DGCBio]

Hi @DLightstone,

Thanks again for the reply. For the case I have posted on, the issue is not consistent. For many people on my team, the elements all appear correctly, while some others see this issue and cannot run our tests. One thing we have noticed, although it is anecdotal, is that it seems that machines running Windows 10 1903 seem to see issues like this more commonly than those running 1809. Again, this is anecdotal.

After installing a build of our app, I tested this behavior 3 times, the first 2 times the elements showed up as expected in the UIAutomation tree. The third time the elements were not there, but in all 3 cases the application looked exactly the same to the user.

[DLightstone]

@DGCBio

Not consistent on the same computer suggests a race condition between the application under test and the test procedure or perhaps something related to task scheduling

One thing to consider is that controls can be created dynamically. This I observe to occur with the TreeView control. If this is a possibility capture the XML during the test just before you expect the missing controls to be available.

[DGCBio]

HI @DLightstone,

We can reproduce the issue manually as well by starting the application and simply viewing the UIAutomation tree with inspect. Therefore, if there is a race condition, it seems to be within Windows involving adding elements of the application to the UIAutomation tree.
I have already captured the details of the UIAutomation tree when the condition occurs, and when it does not in my original post.

Thanks.

[DGCBio]

Some updates to add:

1. This continues to be an important issue for our team. It is critical we resolve it or it will be a showstopper for us being able to use WinAppDriver for test automation (we have been building our UWP automation around WinAppDriver for the last couple years).

2. We had a machine (tablet) that was running Windows 10 1809 and not once did we observe this problem (it was our last machine capable of running our automated tests reliably). This morning it was updated to Windows 10 1903. After the upgrade, this bug can now be reproduced reliably. Again, this issue does not happen 100% of the time, but for most of our Windows 10 1903 machines and tablets it happens more often than not. We had already been harboring suspicion that it was related to some change in the Windows 1903 update, and this seems to confirm it. Perhaps there was an optimization change in the 1903 update that affected the UI Automation server? That is a guess.

3. We have explored:

• Altering the value for the control's AutomationProperties.AccessibilityView property.
• Ensuring all updates for Windows 10 1903 have been downloaded and installed.
• Ensuring all devices are running WinAppDriver 1.1.3.0.
• Attempting to debug the issue using the Accessibility Insights tool.
• Testing across many devices with different operating system versions.

We have also found the following WinAppDriver issues that appear to be related or at least very similar.

Issue 404 Issue 757 Issue 542 Issue 605

[DLightstone]

@DGCBio

With respect to capturing XML of dynamically created controls

Perhaps dynamically created controls can be precluded. Perhaps your capture strategy considered the possibility. I was just trying to indicate a potential observe-ability problem. Inspect and Accessibility Insights will not observe such conditions. They are basically samplers and the sample has to be taken at the appropriate time.

As an example to clarify my intent

If you capture the XML of an application containing a treeview control immediately before and during a treenode relabel you will observe (1) the absence of an edit control in the XML captured immediately before the relabel (2) the presence of an edit control in the XML capture created during the relabel.

With respect to possible bugs I definitely have observed defects related to adding elements to the UIAutomation tree, so this is not something that can be precluded.

Although is not likely to be the same situation with which you are presented. I have a build where TreeItemControls are converted to TreeControls in one project configuration, and properly recognized an a different project configuration.

My bet is Microsoft Bug

[kelly987654]

Does WinAppDriver use the Accessibility dlls in the Net Framework installation? I noticed that Windows 1809 includes Net Framework 4.8.3761, and updating to 1903 downgrades the Net Framework version to 4.8.3572.

Could this be the issue? If so, how can I manually update our 1903 tablet to check if the newer Net Framework version would fix the issue? I tried manually installing Net Framework 4.8 from Microsoft, but it just says I already have 4.8 installed and does nothing.

[DLightstone]

Potentially risky, but you can Use Programs and Features to delete the framework and then reinstall it.

https://dotnet.microsoft.com/download/dotnet-framework/net48

[kelly987654]

Hi @DLightstone ,

Thanks for the response. Net Framework isn't available in the Programs in Features (Control Panel or Win10) to uninstall. I also tried doing a repair and uninstall using their installer.exe with the /repair option which reported success but didn't actually uninstall the framework.

[DLightstone]

@kelly987654

I am running Windows 10 Pro Navigate to settings\Apps (upper right corner) Click on Programs and Features

See upper right corner of image below

[kelly987654]

Hi @DLightstone ,

Thanks, I'm able to find Programs and Features, but I don't see Net Framework within that list.

I think only the Net Framework SDKs appear in the list (not the NET Framework installed with the Windows update). I did try installing them to see if it would help, but it didn't affect the versions in the C:\Windows\Microsoft.NET folder, and when I uninstalled every NET Framework entry from the Programs and Features, the C:\Windows\Microsoft.NET was still there with 4.8 installed.

[kelly987654]

Some more info: The issue is seen ~95% of the time if the app is opened by WinAppDriver using the POST 'session' command. I am not (so far) seeing the issue if I manually open the app prior to starting the WinAppDriver session.

[DGCBio]

@hassanuz we are hoping for a comment from you or someone on your team.

[DLightstone]

@DGCBio

Don't hold your breath. For reasoning, see the book, Crossing the Chasm by Geoffrey Moore

Microsoft's goals are to preempt competition by getting firms to adopt a technology prior to it being fully implemented.

You can see this by looking for the tag "Help Wanted", and evaluating whether it is possible for someone other than a Microsoft employee to be able to provide the help

[hassanuz]

@DGCBio and @kelly987654, can you try contacting me directly at hauraize@microsoft.com.

There's not much we can do from the WinAppDriver side given the context, but it would be worthwhile looking into seeing why the app isn't correctly attributing the UI elements - if this is a .NET framework issue, then we'd want to contact the owners for that to resolve this.

[jeremeguenther]

Was there ever any resolution to this?

I have a similar scenario where I am trying to detect the download bar in Chrome using WinAppDriver. Chrome is started by Selenium, and the WinAppDriver simply grabs the process.

On the development machine WinAppDriver is able to correctly find the download bar every time. However, on a fully updated and patched separate machine WinAppDriver is never able to find the download bar.

[oatsoda]

@jeremeguenther I'd be interested to know also. I am having the same issue with Chrome Extensions.

I have had some success though with passing --force-render-accessibility into chrome on startup.

That's made things render similar to my machine but it is no populating all the automationId properties I expected (for the HTML inside the Chrome popup).