search

nvaccess / nvda

NVDA, the free and open source Screen Reader for Microsoft Windows
Other
2.07k stars 624 forks source link

Windows 11 Voice Access: NVDA does not announce notifications such as microphone active/sleep status #16862

Open josephsl opened 1 month ago

josephsl commented 1 month ago

Background:

Windows 11 22H2 (2022 Update/build 22621) introduces Voice Access, a tool to let people dictate text and perform commands using their voices. While Voice Access is active, an overlay window is displayed with controls such as dictated text, microphone state toggle button, and a help button.

Voice Access uses UIA notification event to notify assistive technologies about changes such as microphone state and dictated text. At the moment NVDA does not announce these notifications because it cannot locate the object in question as seen in the log fragment below. Interestingly, NVDA can report notifications if Narrator is also active.

Steps to reproduce:

Prerequisities: Windows 11 22H2 or later on a computer with a microphone.

  1. If not done so, open Voice Access and download the speech model when prompted.
  2. With Voice Access set up and NVDA running, move system focus to Voice Access window, go to microphone button and press Enter to toggle it.
  3. Alternatively, say "microphone sleep" or "microphone active" (or similar commands to toggle the microphone).

Actual behavior:

The following assumes Voice Access window is focused:

Narrator active NVDA behavior
Yes NVDA announces Voice Access notifications
No NVDA does not announce notifications

 ### Expected behavior: NVDA announces Voice Access notifications while focused on the Voice Access window without Narrator running. The issue of NVDA not announcing UIA notifications from places other than Voice Access can be dealt with as it has to do with focused app check in default UIA notification event handler in NVDA.

NVDA logs, crash dumps and other attachments:

Context: microphone was toggled from Voice Access by pressing Enter.

Input: kb(laptop):enter DEBUG - UIAHandler.UIAHandler.IUIAutomationNotificationEventHandler_HandleNotificationEvent (19:02:28.029) - Dummy-4 (13356): handleNotificationEvent called with notificationKind ActionCompleted, notificationProcessing MostRecent, displayString None, activityID Windows.Shell.VoiceInput.StoppedListening, for element Loading... text with automationID SpeechTextBlock, className TextBlock and frameworkID XAML DEBUG - NVDAObjects.UIA.UIA.init (19:02:28.029) - Dummy-4 (13356): No windowHandle for UIA NvDAObject. Searching UIA element ancestry for nearest windowHandle DEBUG - UIAHandler.UIAHandler.getNearestWindowHandle (19:02:28.029) - Dummy-4 (13356): Locating nearest ancestor windowHandle for element Loading... text with automationID SpeechTextBlock, className TextBlock and frameworkID XAML DEBUGWARNING - UIAHandler.UIAHandler.getNearestWindowHandle (19:02:28.030) - Dummy-4 (13356): error walking up to an element with a valid windowHandle Traceback (most recent call last): File "UIAHandler__init.pyc", line 1338, in getNearestWindowHandle _ctypes.COMError: (-2147467259, 'Unspecified error', (None, None, None, 0, None)) DEBUGWARNING - NVDAObjects.call (19:02:28.030) - Dummy-4 (13356): Invalid NVDAObject: no windowHandle Traceback (most recent call last): File "NVDAObjects__init.pyc", line 96, in call File "NVDAObjects\UIA\init.pyc", line 1553, in init NVDAObjects.InvalidNVDAObject: no windowHandle DEBUG - UIAHandler.UIAHandler.IUIAutomationNotificationEventHandler_HandleNotificationEvent (19:02:28.030) - Dummy-4 (13356): HandleNotificationEvent: Ignoring because no object: NotificationProcessing=3 displayString=None activityId=Windows.Shell.VoiceInput.StoppedListening DEBUG - UIAHandler.UIAHandler.IUIAutomationNotificationEventHandler_HandleNotificationEvent (19:02:28.030) - Dummy-4 (13356): handleNotificationEvent called with notificationKind ActionCompleted, notificationProcessing ImportantAll, displayString Microphone sleep, activityID Windows.Shell.VoiceAccess, for element Microphone sleep button with automationID MicrophoneButtonControl, className Button and frameworkID XAML DEBUG - NVDAObjects.UIA.UIA.init (19:02:28.030) - Dummy-4 (13356): No windowHandle for UIA NvDAObject. Searching UIA element ancestry for nearest windowHandle DEBUG - UIAHandler.UIAHandler.getNearestWindowHandle (19:02:28.030) - Dummy-4 (13356): Locating nearest ancestor windowHandle for element Microphone sleep button with automationID MicrophoneButtonControl, className Button and frameworkID XAML DEBUGWARNING - UIAHandler.UIAHandler.getNearestWindowHandle (19:02:28.032) - Dummy-4 (13356): error walking up to an element with a valid windowHandle Traceback (most recent call last): File "UIAHandler\init__.pyc", line 1338, in getNearestWindowHandle _ctypes.COMError: (-2147467259, 'Unspecified error', (None, None, None, 0, None)) DEBUGWARNING - NVDAObjects.call (19:02:28.032) - Dummy-4 (13356): Invalid NVDAObject: no windowHandle Traceback (most recent call last): File "NVDAObjects\init.pyc", line 96, in call File "NVDAObjects\UIA\init.pyc", line 1553, in init__ NVDAObjects.InvalidNVDAObject: no windowHandle DEBUG - UIAHandler.UIAHandler.IUIAutomationNotificationEventHandler_HandleNotificationEvent (19:02:28.032) - Dummy-4 (13356): HandleNotificationEvent: Ignoring because no object: NotificationProcessing=0 displayString=Microphone sleep activityId=Windows.Shell.VoiceAccess

System configuration

NVDA installed/portable/running from source:

Installed

NVDA version:

alpha-32918,3aafc821

Windows version:

Windows 11 24H2 preview (build 26100.1150)

Name and version of other software in use when reproducing the issue:

Voice Access 24H2 (10.0.26100)

Other information about your system:

Used as a development workstation, subscribed to Windows Insider/release preview channel

Other questions

Does the issue still occur after restarting your computer?

Yes

Have you tried any other versions of NVDA? If so, please report their behaviors.

Reproducible in 2024.1, 2024.2, 2024.3 beta

If NVDA add-ons are disabled, is your problem still occurring?

Yes

Does the issue still occur after you run the COM Registration Fixing Tool in NVDA's tools menu?

Not applicable

Additional context:

I suspect something odd is happening when fetching UIA element window handle while trying to handle notification event form Voice Access (NVDA needs window handle when constructing objects). At least resolving this will allow NVDA to announce things while focused on Voice Access. Getting NVDA to announce Voice Access status from everywhere can be simple as creating an app module to announce notification display string and no more (because focused app is not Voice Access, thus NVDA will ignore notification event by default).

The biggest impact is that NVDA users will be able to use Voice Access more fully, including getting notificatoins about microphone toggle, error alerts, among others. I provided primitive support for Voice Access (at least letting NVDA announce microphone toggle from the keyboard) via an add-on, but resolving the root issue (UIA notification event handling) would be a more effective solution.

Thanks.

josephsl commented 1 month ago

Blocked by #16871. Just like voice typing, I'm thinking about letting people hear Voice Access messages from everywhere.

Thanks.