microsoft / WindowsAppSDK

The Windows App SDK empowers all Windows desktop apps with modern Windows UI, APIs, and platform features, including back-compat support, shipped via NuGet.
https://docs.microsoft.com/windows/apps/windows-app-sdk/
MIT License
3.8k stars 320 forks source link

Add Win32 API support for the GazeHID driver #432

Open JuliusSweetland opened 3 years ago

JuliusSweetland commented 3 years ago

Proposal: Enable access to the GazeInputSourcePreview messages via a Win32 API

Summary

Expose all GazeInputSourcePreview messages (https://docs.microsoft.com/en-us/uwp/api/windows.devices.input.preview.gazeinputsourcepreview?view=winrt-19041) via a Win32 API

Rationale

  1. Support for Win32 apps in general is required, rather than requiring people to rewrite their app to fit UWP. This is the main priority since it is just getting what is available in the UWP API to work in Win32.
  2. Gaze data is required for any part of the screen, not just the active window. Only the active window is supported as of now in the UWP API. For example, my project 'Optikey' requires access to gaze data at any point on the screen, not just where the UWP application is the target.

Scope

The ability to receive x,y screen co-ordinates for the current gaze location in a Win32 application from any point on the screen.

JayToltTech commented 3 years ago

I think this is a great idea. As an ISV who writes eye gaze enabled accessibility software, I need to use traditional desktop development in order to handle complex cross-application interactions (e.g. controlling other processes, windows management, input injection).

I was sad that Microsoft introduced a new API for eye gaze, but restricted it to UWP apps -- I couldn't take advantage of it. So I've had to maintain a fragile 'interop layer' consuming four different eye tracker hardware vendor SDKs and abstract them away so my app could work with any of the major eye trackers. What a pain! On top of this, each of the APIs shifts every year or so and each of them comes with individual legal terms and complications.

This is a huge burden for an assistive technology ISV to handle. Looking forward to consideration for this request, I think this will really help Microsoft stay on top of the assistive technology industry.

aborzin commented 3 years ago
  1. The restriction of the UWP app is really painful and it would be great to have it in the Win32 API.
  2. Having the gaze data outside of the active window is critical for many usages. For some of the usages, having the gaze data outside of the physical screen boundaries is very important too.
feisu commented 3 years ago

Thank you for the proposal. We are looking into how to enable the Gaze API for win32 apps. Stay tuned!

whitmer commented 3 years ago

I write a cross-platform communication app for people with disabilities, sort of like OptiKey but more cloud-focused. Because I'm a single developer supporting multiple platforms, I develop a web-based solution and then wrap it in native libraries (Cordova and Electron). Because I'm using Electron, I can't build a UWP app that would get access to the gaze APIs. My only alternative is to maintain a wrapper library that talks to all the individual vendor-specific libraries instead, which is obviously more fragile and less robust than a single standardized API, and I worry about its long-term viability. I am also very hopeful we can get win32 access to the gaze APIs in a standardized way.

Thanks for all your work on this!

riverar commented 3 years ago

@whitmer Some outdated info there, which might be good news for you -- you don't need to be a UWP app to call (most) WinRT APIs. And you can certainly package your Electron app as a UWP app.

jonwis commented 3 years ago

The core problem here is something we're trying to resolve across the API surface - many WinRT APIs need window display information, like where on screen they should pop up additional UX, a thread whose dispatcher should be used, or a visual/window whose input should be processed. APIs with names like GetForCurrentView or RequestAsync or GetForCoreWindow have an implicit dependency on the CoreWindow type. GazeInputSourcePreview.GetForCurrentView is like that.

As part of #370 we're talking about how to bridge the windowing gap, and part of that will be adding overloads or interop helpers like GazeInputSourcePreview.GetForWindowId(WindowId) so you can specify which window to target, rather than us inferring one from the environment.

We've also changed our API design guidelines to prefer "for explicit window" rather than "for implicit window" shapes. Stay tuned as we work through those!

Some of these WinRT types do support "initialize with specific HWND" behavior already - check out https://github.com/AdamBraden/WindowsInteropWrappers for a pre-made set of helpers that do it. GazeInputSourcePreview does not have an interop wrapper yet. @feisu will ping the thread when we get support ready for it.

kmcnaught commented 3 years ago

I just wanted to add my support for this - I write software based on Optikey that provides eye-gaze controls for PC gaming, and we're currently beholden to third party APIs for each eye tracker, which adds development effort and is fragile in the face of the changing commercial landscape. Moving forward I'd love to see the new HID standard more widely adopted, but it would need Win32 support to be used in these applications.

dennyronngren commented 3 years ago

This is a great request. The gaze input in UWP are great, but a piece missing is their availability in win32, which limits their potential for being a generic building block to build gaze enabled applications, since in practice not all developers are targeting UWP.

The big step forward in the request would be for Windows to handle the routing of eye gaze to the correct window, regardless if it's a core window or not. As it is now, to be able to use gaze in win32 apps (either our own or through our SDKs), we (Tobii) are forced to jump through hoops to understand how different windows are overlapping, their transparency, etc. to be able to map gaze to the right window. This is very fragile when there's a change in the OS.

It would be far simpler and better if we could rely on Windows doing this routing, which should be possible if gaze API would be available in win32 in addition to the universal platform.