Elements retrieved with `ControlFromCursor` cannot be retrieved with `Control` as well

[cmin764]

For example we can find Notepad's Text editor element name with the cursor's control retrieval, but Control(Name="Text editor") won't be able to find the element at all. The very same element appears in the Accessibility Insights tool as well but isn't visible when walking through the element tree (children traversal).

Notepad version (on Windows 11 ARM through Parallels in Mac M1):

Windows Notepad 11.2302.16.0
© 2023 Microsoft. All rights reserved.

[yinkaisheng]

I have met this issue when automate another app(not on ARM Windows) in the past, but I don't know how to fix. System API can't get the control through the element tree, ControlFromCursor can get it.

[fabioz]

I just stumbled upon this with notepad too. From what I can tell, the issue is when DesktopWindowXamlSource is found.

From the docs: https://learn.microsoft.com/en-us/uwp/api/windows.ui.xaml.hosting.desktopwindowxamlsource?view=winrt-22621 it says that:

Enables a non-UWP desktop application (for example, a WPF or Windows Forms application) to host UWP controls in any UI element that is associated with a window handle (HWND).

So, the issue seems to be bridging among the non-UWP/UWP.

The structure when searching for children is something as:

Code:

def print_children(el, level=0):
    for c in el.GetChildren():
        print(" " * (level * 2), c)
        print_children(c, level + 1)

control = auto.ControlFromCursor()
print_children(control.GetTopLevelControl())

ControlType: WindowControl    ClassName: Notepad    AutomationId:     Rect: (309,158,1749,911)[1440x753]    Name: '*LocalizedControlType - Notepad'    Handle: 0x4F090E(5179662)
 ControlType: PaneControl    ClassName: Windows.UI.Composition.DesktopWindowContentBridge    AutomationId:     Rect: (355,168,642,200)[287x32]    Name: 'DesktopWindowXamlSource'    Handle: 0x9908D4(10029268)
   ControlType: PaneControl    ClassName: Windows.UI.Input.InputSite.WindowClass    AutomationId:     Rect: (355,168,642,200)[287x32]    Name: ''    Handle: 0x310908(3213576)
     ControlType: TabControl    ClassName: Microsoft.UI.Xaml.Controls.TabView    AutomationId: Tabs    Rect: (355,168,642,200)[287x32]    Name: ''    Handle: 0x0(0)
       ControlType: ListControl    ClassName: ListView    AutomationId: TabListView    Rect: (357,168,607,200)[250x32]    Name: ''    Handle: 0x0(0)
         ControlType: TabItemControl    ClassName: ListViewItem    AutomationId:     Rect: (359,168,605,200)[246x32]    Name: 'LocalizedControlType. Modified.'    Handle: 0x0(0)
           ControlType: TextControl    ClassName: TextBlock    AutomationId:     Rect: (372,176,491,191)[119x15]    Name: 'LocalizedControlType'    Handle: 0x0(0)
           ControlType: TextControl    ClassName: TextBlock    AutomationId:     Rect: (574,178,586,190)[12x12]    Name: '\ue827'    Handle: 0x0(0)
       ControlType: ButtonControl    ClassName: Button    AutomationId: AddButton    Rect: (610,173,642,197)[32x24]    Name: 'Add New Tab'    Handle: 0x0(0)
         ControlType: TextControl    ClassName: TextBlock    AutomationId:     Rect: (620,179,632,191)[12x12]    Name: '\ue710'    Handle: 0x0(0)
     ControlType: PaneControl    ClassName: Microsoft.UI.Xaml.Controls.TeachingTip    AutomationId: TeachingTip    Rect: (0,0,0,0)[0x0]    Name: 'Notepad automatically saves your progress. All your content will be available the next time you open Notepad.'    Handle: 0x0(0)
 ControlType: PaneControl    ClassName: Windows.UI.Composition.DesktopWindowContentBridge    AutomationId:     Rect: (315,200,1743,233)[1428x33]    Name: 'DesktopWindowXamlSource'    Handle: 0x11D05B6(18679222)
   ControlType: PaneControl    ClassName: Windows.UI.Input.InputSite.WindowClass    AutomationId:     Rect: (315,200,1743,233)[1428x33]    Name: ''    Handle: 0x290FA4(2690980)
     ControlType: MenuBarControl    ClassName:     AutomationId: MenuBar    Rect: (315,200,474,232)[159x32]    Name: ''    Handle: 0x0(0)
       ControlType: MenuItemControl    ClassName: Microsoft.UI.Xaml.Controls.MenuBarItem    AutomationId:     Rect: (319,200,360,232)[41x32]    Name: 'File'    Handle: 0x0(0)
         ControlType: ButtonControl    ClassName: Button    AutomationId: ContentButton    Rect: (319,200,360,232)[41x32]    Name: 'File'    Handle: 0x0(0)
           ControlType: TextControl    ClassName: TextBlock    AutomationId:     Rect: (329,207,350,226)[21x19]    Name: 'File'    Handle: 0x0(0)
       ControlType: MenuItemControl    ClassName: Microsoft.UI.Xaml.Controls.MenuBarItem    AutomationId:     Rect: (368,200,412,232)[44x32]    Name: 'Edit'    Handle: 0x0(0)
         ControlType: ButtonControl    ClassName: Button    AutomationId: ContentButton    Rect: (368,200,412,232)[44x32]    Name: 'Edit'    Handle: 0x0(0)
           ControlType: TextControl    ClassName: TextBlock    AutomationId:     Rect: (378,207,402,226)[24x19]    Name: 'Edit'    Handle: 0x0(0)
       ControlType: MenuItemControl    ClassName: Microsoft.UI.Xaml.Controls.MenuBarItem    AutomationId:     Rect: (420,200,470,232)[50x32]    Name: 'View'    Handle: 0x0(0)
         ControlType: ButtonControl    ClassName: Button    AutomationId: ContentButton    Rect: (420,200,470,232)[50x32]    Name: 'View'    Handle: 0x0(0)
           ControlType: TextControl    ClassName: TextBlock    AutomationId:     Rect: (430,207,460,226)[30x19]    Name: 'View'    Handle: 0x0(0)
     ControlType: ButtonControl    ClassName: Button    AutomationId: SettingsButton    Rect: (1705,201,1735,231)[30x30]    Name: 'Settings'    Handle: 0x0(0)
 ControlType: PaneControl    ClassName: Windows.UI.Composition.DesktopWindowContentBridge    AutomationId:     Rect: (315,873,1743,905)[1428x32]    Name: 'DesktopWindowXamlSource'    Handle: 0x82081A(8521754)
   ControlType: PaneControl    ClassName: Windows.UI.Input.InputSite.WindowClass    AutomationId:     Rect: (315,873,1743,905)[1428x32]    Name: ''    Handle: 0x270EFC(2559740)
     ControlType: TextControl    ClassName: TextBlock    AutomationId: ContentTextBlock    Rect: (330,882,393,898)[63x16]    Name: 'Line 7,\nColumn 9'    Handle: 0x0(0)
     ControlType: TextControl    ClassName: TextBlock    AutomationId: ContentTextBlock    Rect: (1374,882,1406,898)[32x16]    Name: 'Zoom'    Handle: 0x0(0)
     ControlType: TextControl    ClassName: TextBlock    AutomationId: ContentTextBlock    Rect: (1462,882,1553,898)[91x16]    Name: ' Windows (CRLF)'    Handle: 0x0(0)
     ControlType: TextControl    ClassName: TextBlock    AutomationId: ContentTextBlock    Rect: (1622,882,1658,898)[36x16]    Name: ' UTF-8'    Handle: 0x0(0)
 ControlType: TitleBarControl    ClassName:     AutomationId:     Rect: (0,0,0,0)[0x0]    Name: ''    Handle: 0x0(0)
   ControlType: MenuBarControl    ClassName:     AutomationId: MenuBar    Rect: (317,166,339,188)[22x22]    Name: 'System'    Handle: 0x0(0)
     ControlType: MenuItemControl    ClassName:     AutomationId:     Rect: (317,166,339,188)[22x22]    Name: 'System'    Handle: 0x0(0)
   ControlType: ButtonControl    ClassName:     AutomationId:     Rect: (1604,159,1651,189)[47x30]    Name: 'Minimize'    Handle: 0x0(0)
   ControlType: ButtonControl    ClassName:     AutomationId:     Rect: (1651,159,1697,189)[46x30]    Name: 'Maximize'    Handle: 0x0(0)
   ControlType: ButtonControl    ClassName:     AutomationId:     Rect: (1697,159,1744,189)[47x30]    Name: 'Close'    Handle: 0x0(0)

And the structure from picking an element at a given position and then going through the parents is:

Code:

def print_parents(c):
    print(c)
    p = c.GetParentControl()
    if p:
        print_parents(p)

control = auto.ControlFromCursor()
print_parents(control)

ControlType: DocumentControl    ClassName: RichEditD2DPT    AutomationId:     Rect: (315,233,1743,873)[1428x640]    Name: 'Text editor'    Handle: 0x2512A4(2429604)
ControlType: PaneControl    ClassName: NotepadTextBox    AutomationId:     Rect: (315,233,1743,873)[1428x640]    Name: ''    Handle: 0x2E123A(3019322)
ControlType: WindowControl    ClassName: Notepad    AutomationId:     Rect: (309,158,1749,911)[1440x753]    Name: '*LocalizedControlType - Notepad'    Handle: 0x4F090E(5179662)
ControlType: PaneControl    ClassName: #32769    AutomationId:     Rect: (0,0,1920,1080)[1920x1080]    Name: 'Desktop 1'    Handle: 0x10010(65552)