Windows 11 Design Elements Are Not Reported by NVDA When Mouse Tracking Is On

[ferhatbkr]

Steps to reproduce:

• Turn on mouse tracking in NVDA.
• Hover mouse cursor over taskbar items. In the same way, navigate the toolbar buttons of Windows Explorer, the top menus of the New Notepad application with the mouse.

  Actual behavior:

  NVDA will just say "DesktopWindowXamlSource" instead of naming items. The same is true for all applications developed with Windows 11's new Flat Design. Like PowerToys that don't come pre-installed with Windows. Narrator items read their names as expected.

  Expected behavior:

  NVDA must accurately report information on items, as Narrator does.

  System configuration

  NVDA installed/portable/running from source:

  Installed

  NVDA version:

  NVDA 2021.3.4

  Windows version:

  Windows 11 Version 10.0.22000.588 Build 22000.588

  Name and version of other software in use when reproducing the issue:

Other information about your system:

Other questions

Does the issue still occur after restarting your computer?

Yes

Have you tried any other versions of NVDA? If so, please report their behaviors.

If NVDA add-ons are disabled, is your problem still occurring?

Yes

Does the issue still occur after you run the COM Registration Fixing Tool in NVDA's tools menu?

Yes

[josephsl]

Hi,

Can you tell us things Narrator says so we can investigate what's going on?

Technical: these design elements expose class names as their label (name).

Thanks.

[josephsl]

Hi,

Also, note that design elements are not really the same thing as item labels, and I think that could confuse people.

Thanks.

[Molitvan]

I have the same problem. Also, I noticed that its happening with almost all new ms apps like windows terminal for example. I assume that some library used in these apps/elements is making an issue.

Also, I think there is a way to fix the problem with taskbar. Actualy, in windows 11, there is some kind of a cover on top of the taskbar. This cover makes nvda read that instead of the actual taskbar when you hover over it. You can test this by simply hovering over the top edge of the taskbar. If there is a way to remove this, the taskbar will work. Also, by removing that cover you will restore the old taskbar context menus :D.

I will update this comment if my theory about the taskbar turns out to be true

[ferhatbkr]

Hi,

Can you tell us things Narrator says so we can investigate what's going on?

Technical: these design elements expose class names as their label (name).

Thanks.

The name or tooltip of the Start Menu, pinned applications, and system tray items (including date and time) in the Taskbar is not reported by NVDA.

In Windows Explorer, buttons and menus on the bar that Narrator calls "Application Bar" (ex: new, cut, copy, rename buttons and sort, view menus) are not spoken.

I said "design element" because I had this problem in applications updated according to this design, which came with the new design of Widnows 11. My fault.

[josephsl]

Hi,

About terminology: yep, I can see why it can be confusing.

Also another technical note: until Windows 10 at least shell elements such as Start button label was exosed as a MSAA (Microsoft Active Accessibility) objects, whereas they are seen as UIA (UI Automation) objects in Windows 11.

Thanks.

[josephsl]

Hi,

Dramatic turn: this explains why touch hover gesture (hold down one finger while navigating the screen) is not effective in Windows 11 taskbar - no item announcement when one hovers around the bottom of the screen. I will raise a separate issue for touchscreen interaction as looks like these might be related.

Thanks.

[josephsl]

Hi,

April 2022 update: telling NVDA to treat DesktopWindowXamlSource objects as UIA fixes it for BOTH mouse and touch interaction. Note that there are multiple elements with this "label", so the easy fix will require informing the File Explore app module that these elements are indeed UIA objects. I'll come up with a prototype fix via Windows App Essentials add-on if possible (possibly later today which will require changes to File Explorer app module).

Thanks.

[josephsl]

Hi,

Update: yes, UIA element reclassification fixes it. For folks wishing to write a pull request (I can do so next week if needed), the following window class names must be checked from isGoodUIAWindow method in File Explorer app module class: MSTaskSwWClass (parts of taskbar), TrayNotifyWnd (systray list), ToolbarWindow32 (toolbar), ReBarWindow32 (parts of taskbar), Shell_TrayWnd (taskbar). Of course you must make sure this is Windows 11 or later. I'm sure there is a much better solution.

Thanks.

[josephsl]

Hi,

For Windows 11 Notepad, a window with the class name of "Notepad" shuold be treated as good UIA window. Because of this, a solution that works across programs is ideal.

Additional findings:

1. DesktopWindowXamlSource class name is used by classic Win32 apps to host XAML controls.
2. A more effective solution is detecting these windows from UIA handler thread itself and telling NVDA to treat these as good UIA windows. While editing isGoodUIAWindow method from app modules would work, it isn't generalizable across apps as the class name in question (see above) is found across multiple apps.
3. No separate issue is required to describe touchscreen interaction behavior as the issue described here impacts both mouse and touch. Therefore a pull request that fixes this problem will affect both interaction scenarios at once.
4. As a workaround, enable "report tooltips" from object presentation settings panel and enable "follow mouse cursor" checkbox from review cursor settings panel.
5. Although the computer used to test possible solutions is running a nickel build (22598 to be exact), a pull request for this issue will also work on Windows 11 Version 21H2 (build 22000) as evidenced by the fact that the described workaround and potential solutions are working in a virtual machine running the said build.

As for pull requests, while I may not be able to come up with a solution that works across apps (I'm exhausted), I can do something about at least File Explorer (Windows 11 Notepad will be handled via Windows App Essentials for now).

CC @MichaelDCurran in case you heard something from XAML team inside Microsoft.

Thanks.

[lukaszgo1]

Just out of interest I've looked into this briefly and I'm not understanding something here. Given that NVDA correctly decides to use UIA for these controls when accessed via keyboard why the same fails when done via mouse / touch screen. In other words why objectFromPoint returns the MSAA hosting window rather than the real UIA object?

[josephsl]

Hi, that’s a good question – I thought about experimenting with object from point function to see if underlying element can be fetched, but figured that it may as well have to do with the UI framework in use. By the way, any updates from the person who wanted to see if “removing a cover” from the screen might help with this situation?

[mzanm]

I'm not sure if this is really possible or not, but can't any window with the role of unknown or pane and name of "DesktopWindowXamlSource" be considered a good UIA window? I don't think adding window class names will work, as a name needs to be added for each app, for example mouse navigation on windows terminal is broken and it has the window lass name of "CASCADIA_HOSTING".

[josephsl]

Hi, in theory, yes but remember that these windows host XAML/UIA elements while they themselves are part of classic apps (some apps use UIA while others are MSAA universes). The most effective way (besides asking for WinUI changes) is gathering a unique data from these windows as part of UIA handler thread and telling NVDA right then that these are good windows (see source/UIAHandler/init.py/UIAHandler class for details). As noted in the Windows App Essentials I linked to in this issue, I’m taking a two-pronged approach (global plugin and app module method) until a more general solution is found. Thanks.

[josephsl]

Hi,

UIA reclassification method cannot be used with Windows Terminal - if I do it, when the mouse moves to terminal window, the app crashes. This is observed with both global plugin and app module method.

Thanks.

[josephsl]

Hi,

Expect a pull request to resolve part of this issue soon - likely limited to cover shell elements such as Taskbar as this is one of the most prominent areas of the operating system and likely to be navigated with mouse and/or touch. Notepad and File Explorer command bar (that's what they are really) may or may not be covered.

P.S. In the upcoming pull request, File Explorer app module will state that a more permanent solution be found and my workaround removed once that happens.

Thanks.

[josephsl]

Hi,

May 2022 update: until a general solution is found, for File Explorer/shell elements, the following solutions are available that involves good UIA window method:

1. Selective class names: provide a list of window class names representing various shell elements such as taskbar, notification area, and device-specific items such as touch keyboard button (check with user32.dll::GetClassName(hwnd) wrapped inside winUser module). While the solution is very specific, additional class names must be added once discovered - if not now, perhaps in future Windows 11 updates.
2. Ancestor checks: check ancestor (mostly top-level_ windows by calling user32.dll::GetAncestor function with GA_ROOT (2) as ancestor parameter (this function is wrapped as winUser.getAncestor(hwnd)). This allows any element hosted (child of) by a limited number of top-level windows to be recognized and navigated to at the cost of performance as Windows (specifically, window manager) must navigate GUI hierarchy.
3. Go all in on UIA except for one or two class names: in other words, all File Explorer windows should be classified as UIA elements except for ones requiring IAccessible implementation (mostly ones with odd UIA server-side implementation such as desktop icons/SysListView32). While not as specific as the above two solutions, only a limited set of class names are checked which helps in performance but may expose issues with UIA elements so a cautionary note must be added.

Even with one or more of these solutions implemented, the pull request MUST include a note that a more general solution is ideal to cover other programs such as Notepad and Windows Terminal. I'm leaning toward either a second or a third solution unless folks believe the first solution is better or offer something completely different (including unexpected solutions).

In summary:

1. Reclassifying windows that host a window named "DesktopWindowXamlSource" as UIA windows partially solves the problem (causes a crash in Windows Terminal according to tests, hence a partial solution).
2. DesktopWindowXamlSource windows are found in Win32 (desktop) apps that host XAML controls. In Windows 11, this is prominent if an app uses WinUI framework. These windows are found not only in File Explorer, but also in apps such as Windows 11 Notepad and Windows Terminal, so likely a hybrid solution involving app modules and accessibility API handlers (and even NVDA objects) is the best.
3. This issue is not present in Windows 10, thus the solutions should target at least Windows 11 Version 21H2 (RTM/build 22000).
4. Solving mouse navigation problem for DesktopWindowXamlSource windows can resolve touchscreen interaction with controls hosted within them. Therefore testing solutions with mouse is required (and touch is recommended if the computer is equipped with a touchscreen).

Next steps: within the next 72 hours, a PR will be submitted which will implement the third solution unless folks prefer something else in the meantime.

Thanks.

[XLTechie]

@josephsl while this issue is somewhat outside my area of study, a thought occurs.

Your solution 2, in the last comment, seems like, from your description, to be the least likely to come up with false results. However it is slow. Would some sort of caching be possible here? An in-memory cache of successes and failures, to prevent having to back-walk the tree for every control after it is once known?

[josephsl]

Hi, caching doesn’t help (thought about it yesterday) as isGoodUIAWindow relies on a window handle (hwnd), and the handle changes every time a shell element such as Task View opens. There are many documented instances where this is happening, and Windows 11 UI is one of these (recent) cases. We can cache things (or use mechanisms such as LRU (least recently used)) if method arguments remain the same across calls but this is clearly not the case. Thanks.

[josephsl]

Hi,

Upon further testing, solution 2 will be the way to go (no performance issues observed while testing compared to solutions 1 and 3) - implementing solution 3 causes NVDA to say "system" when Alt+Space is pressed instead of it saying "restore" as before (regression). Also, adding isGoodUIAWindow to check for UI design elements will change UIA tree as seen by NVDA (at least observed in Notepad), requiring edits to code that traverses the object hierarchy to locate things; thankfully it doesn't affect File Explorer. Pull request incoming (I may not respond to reviews until late May but a PR will be provided for commenting or solutions to be offered by others).

Thanks.

[josephsl]

Hi,

November 2023 update: I think I found a possibly more general solution: use windowUtils.findDescendantWindow(hwnd, className="Windows.UI.Composition.DesktopWindowContentBridge"). Looks like the issue with Windows Terminal where mouse movement crashes it is gone.

The bigger problem: how do we detect top-level windows and at which point should we check for descendants?

Thanks.