Missing UI Automation

[them0ntem]

The UI Automation API is defined in several different C/C++ header files

Header file	Description
UIAutomationClient.h	Defines the interfaces and related programming elements used by UI Automation clients.
UIAutomationCore.h	Defines the interfaces and related programming elements used by UI Automation providers.
UIAutomationCoreApi.h	Defines general constants, GUIDs, data types, and structures used by UI Automation clients and providers. It also contains definitions for the deprecated node and control pattern functions.
UIAutomation.h	Includes all of the other UI Automation header files. Because most UI Automation applications require elements from all UI Automation header files, it is best to include UIAutomation.h in your UI Automation application projects instead of including each file individually.

MSDN Reference

[DataTriny]

Hi @themontem ,

Actually, I already did all the work in #927 and #928, which are blocked by #883. Maybe you could help with reviewing?

However, and regarding all the COM interface and class definitions, it would be much simpler if we could use macros from microsoft/com-rs because they provide an easy way to create COM classes and implement interfaces for them.

CC @retep998 and @rylev.

[w01fgang]

@DataTriny thank you for your effort! Could you also provide an example of how to use this? For example, how can I call ElementFromHandle? Thank you!

[DataTriny]

Hello @w01fgang ,

Glad you asked because I forgot to include a COM class in my last PR (fixed in #928).

Here is how you can use IUIAutomation::ElementFromHandle to get the title of the foreground window:

use widestring::WideString;
use winapi::{Class, Interface};
use winapi::ctypes::c_void;
use winapi::shared::winerror::FAILED;
use winapi::shared::wtypesbase::*;
use winapi::um::combaseapi::CoCreateInstance;
use winapi::um::objbase::CoInitialize;
use winapi::um::oleauto::{SysAllocString, SysFreeString, SysStringLen};
use winapi::um::uiautomationclient::{CUIAutomation, IUIAutomation, IUIAutomationElement, UIA_HWND};
use winapi::um::winuser::*;

fn main() {
    unsafe {
        if FAILED(CoInitialize(std::ptr::null_mut())) {
            eprintln!("Failed to initialize COM.");
            return;
        }
        let mut p_automation: *mut IUIAutomation = std::ptr::null_mut();
        let hr = CoCreateInstance(
            &CUIAutomation::uuidof(),
            std::ptr::null_mut(),
            CLSCTX_INPROC_SERVER,
            &IUIAutomation::uuidof(),
            &mut p_automation as *mut *mut IUIAutomation as *mut *mut c_void);
        if FAILED(hr) {
            eprintln!("Failed to initialize UI Automation.");
            return;
        }
        let hwnd = GetForegroundWindow() as UIA_HWND;
        let mut p_element: *mut IUIAutomationElement = std::ptr::null_mut();
        let hr = p_automation.as_ref().unwrap().ElementFromHandle(hwnd, &mut p_element as *mut *mut IUIAutomationElement);
        if FAILED(hr) {
            eprintln!("Failed to get UI Automation element for foreground window.");
            return;
        }
        let mut raw_name = SysAllocString(std::ptr::null());
        let hr = p_element.as_ref().unwrap().get_CurrentName(&mut raw_name);
        if FAILED(hr) {
            eprintln!("Failed to get name of foreground window.");
            return;
        }
        let name = WideString::from_ptr(raw_name, SysStringLen(raw_name) as usize);
        SysFreeString(raw_name);
        println!("Name of foreground window: {}", name.to_string_lossy());
    }
}

However, if you would like to implement an UI Automation server, you would have to create new COM classes, and manually instanciate the vtables for pointers to them.

Hope this address your question.

Regards.

[w01fgang]

@DataTriny Great! Thanks a lot!