Inspect.exe,WinAppDriver UI Recorder and WinAppdriver are not able to inspect elements

[mlehinchenko]

We are developing Word plugin which creates a templates of documents. We used inpsect.exe,WinAppDriver UI Recorder and WinAppdriver in code but neither of those are able to discover elements. Bringing an example of template and screenshots with the problem to reproduce the issue.

inner_tables_bug_template.docx

Versions: WinAppdriver 1.1.1 Word 1808 Windows 10

Please let me know what else I can provide to help investing the issue.

[naeemakram]

Maybe these are custom controls. There are some controls which can't be captured by Windows Automation tools. Try clicking the element and press Ctrl + C to copy the text.

[mlehinchenko]

hi @naeemakram if we are talking about clicking with the automation code then this is impossible due to non-visibility of elements. If you mean manually then everything is ok as this is a plugin specially designed to work with it manually. Maybe other ideas?

[naeemakram]

Are you trying to retrieve the value of the control? Do you want to click it?

I remember working with automation of a WinForms application, they were using DexExpress controls. Coded UI was unable to access those controls. We had to buy some stuff from the developers in order to make it work. I was wondering that maybe something similar is going on over here. Otherwise, you may try to handle things with typing various key combinations in your automation code. Like for example I used Ctrl+C in recently to retrieve the values of a grid cell which was undetectable by Coded UI. I knew the outer boundary of the grid, so I clicked 20 pixels below it and checked if any value was present in ClipBoard or not. This too can be a helpful hack in some cases.

Just some food for thought.

 internal string FindGridDocumentsAndClickTenPixelsBelowFirstControl()
        {
            var grDocuments = DocumentsWindow().Find<WinWindow>(new PropertyExpressionCollection { new PropertyExpression(WinClient.PropertyNames.ControlName, "fgDocuments"), new PropertyExpression(WinWindow.PropertyNames.ControlName, "fgResults") });

            Debug.WriteLine($"***** {grDocuments.ToString()}");

            WinClient fgDocuments = new WinClient(grDocuments); 
            fgDocuments.SearchProperties.Add(WinClient.PropertyNames.ControlName, "fgDocuments");

            fgDocuments.DrawHighlight();

            StringBuilder sbCopied = new StringBuilder(100);

            for (int i = 30; i < 400; i+= 15)
            {
                Mouse.Click(new System.Drawing.Point(fgDocuments.BoundingRectangle.X + 10, fgDocuments.BoundingRectangle.Y + i));
                Debug.WriteLine($"***** i: {i}");
                Playback.Wait(1000);
                Keyboard.SendKeys("^C");
                if (Clipboard.ContainsText(TextDataFormat.Text))
                {
                    sbCopied.AppendLine(Clipboard.GetText(TextDataFormat.Text));
                    // Do whatever you need to do with clipboardText
                }
            }

            return sbCopied.ToString();
        }

[timotiusmargo]

Hi @mlehinchenko,

After inspecting the document you attached above, I observed that title<Add Title here> and anything within that right hand side table column are not added into the UIA tree. The table element you highlighted in this screenshot actually refers to the entire main table that also encapsulate the left column.

With the lack of presence in UIA tree, the elements on the right column are not interactable or inspectable using WinAppDriver. Nevertheless, you can certainly update your word plugin (or the UI framework it uses) to populate the UIA tree properly since you own this plugin development. This will also be an opportunity to name each node with a proper AutomationId to make the automation task much easier and more scalable.

[mlehinchenko]

@timotiusmargo do you recommend changing the layout and move the title outside the parent table? If yes, then it's not a case for us. Due to requirements, it must be inside of the table. Is there other way?

[timotiusmargo]

Hi @mlehinchenko,

You don't need to change the UI layout of your application at all. Instead you need to make your plugin accessible (working with UI Automation). The following document may help: https://docs.microsoft.com/en-us/windows/desktop/winauto/uiauto-providerportal

[mlehinchenko]

@timotiusmargo looking into that. Will let you know. Thanks

[shaktim121]

I am working with a WPF stand alone Application where while inspecting with inspect.exe, I am seeing the below error: Cannot get object from point: [Error: FromPoint: hr=0xFFFFFFFF80070005 - Access is denied.]

But it is specific to few applications only. What should be do in this case to get the object properties?

[naeemakram]

Try running the inspect.exe as an Administrator.

[shaktim121]

I don't think that being the issue because our of set of applications, it happens for few of them only. Rest works fine.

[mlehinchenko]

@timotiusmargo We could not make changes to plugin as it involves changes to all the plugin, Is there any other way? Please Advice.

[mlehinchenko]

test2.docx test1.docx We tried other approaches and changed the plugin but still the situation is the same. 1803 WORD works as expected 1808 WORD - Elements cannot be found

I provided a simple examples of tables. @timotiusmargo

[MelnykNikita]

get the same issue @timotiusmargo @mlehinchenko Versions: WinAppdriver 1.1.1 Word 1808 Windows 10

[OleksandrPodoliako]

Hi. I run into the same problem. Do you have any updates regarding this issue?

Versions: WinAppdriver 1.1.1 Word 1808 Windows 10

[YuriStopkin]

I`ve faced same problem. Watching this thread since may. Any updates or solutions?

[oyablonskyi]

Hi. We have faced with the same problem on WinDriver v1.1.1. Layout structure is pretty match the same as @mlehinchenko mentioned.

[sMyslovskyi1984]

Have the same problem. Need to fix as soon as possible

[Derevyaa]

Hi have the same problem with WinDriver v1.1.1. Layout

[alex85vol]

Experiencing the same issue as guys posted before. Are there any updates? My environment: WinAppdriver 1.1.1 Word 1808 Windows 10

[yaroslavkratsylo]

Faced with the same issue with WinDriver v1.1.1.

[DedAntip]

inpsect.exe,WinAppDriver UI Recorder and WinAppdriver are not able to inspect elements Has anyone run into the same problem?

[jenny-yud]

Actually I have also been facing this issue since May with WinDriver v1.1.1. Could anyone provide me with information how to solve it?

[jgarciabu]

Adding my name to this issue as well. Some elements are detectable, but not all. This halts test creation. If I can reliably and consistently detect elements in my app using inspect.exe then winappdriver would be the best solution for our efforts. In this case, inspect sees the object from a higher level, but should be able to see that the object splits out to a dropdown menu beneath it. Please see images for reference.

Visual studio object tree detection

Inspect.exe detection (higher level object detection/does not detect the individual button as visual studio does above)

[naeemakram]

Hi, I've published a course about test automation using WinAppDriver. Given below is a special launch promo code(valid 4 days). Automated UI Testing Windows Apps in Appium WinAppDriver, C#

[BalarajAdappa]

Hi,

Even i am also facing same issue ,some elements are detectable some are not detecting.