Windows 11 22H2 (2022 Update/build 22621) introduces Voice Access, a tool to let people dictate text and perform commands using their voices. While Voice Access is active, an overlay window is displayed with controls such as dictated text, microphone state toggle button, and a help button.
Voice Access uses UIA notification event to notify assistive technologies about changes such as microphone state and dictated text. At the moment NVDA does not announce these notifications because it cannot locate the object in question as seen in the log fragment below. Interestingly, NVDA can report notifications if Narrator is also active.
Steps to reproduce:
Prerequisities: Windows 11 22H2 or later on a computer with a microphone.
If not done so, open Voice Access and download the speech model when prompted.
With Voice Access set up and NVDA running, move system focus to Voice Access window, go to microphone button and press Enter to toggle it.
Alternatively, say "microphone sleep" or "microphone active" (or similar commands to toggle the microphone).
Actual behavior:
The following assumes Voice Access window is focused:
Narrator active
NVDA behavior
Yes
NVDA announces Voice Access notifications
No
NVDA does not announce notifications
### Expected behavior:
NVDA announces Voice Access notifications while focused on the Voice Access window without Narrator running. The issue of NVDA not announcing UIA notifications from places other than Voice Access can be dealt with as it has to do with focused app check in default UIA notification event handler in NVDA.
NVDA logs, crash dumps and other attachments:
Context: microphone was toggled from Voice Access by pressing Enter.
Input: kb(laptop):enter
DEBUG - UIAHandler.UIAHandler.IUIAutomationNotificationEventHandler_HandleNotificationEvent (19:02:28.029) - Dummy-4 (13356):
handleNotificationEvent called with notificationKind ActionCompleted, notificationProcessing MostRecent, displayString None, activityID Windows.Shell.VoiceInput.StoppedListening, for element Loading... text with automationID SpeechTextBlock, className TextBlock and frameworkID XAML
DEBUG - NVDAObjects.UIA.UIA.init (19:02:28.029) - Dummy-4 (13356):
No windowHandle for UIA NvDAObject. Searching UIA element ancestry for nearest windowHandle
DEBUG - UIAHandler.UIAHandler.getNearestWindowHandle (19:02:28.029) - Dummy-4 (13356):
Locating nearest ancestor windowHandle for element Loading... text with automationID SpeechTextBlock, className TextBlock and frameworkID XAML
DEBUGWARNING - UIAHandler.UIAHandler.getNearestWindowHandle (19:02:28.030) - Dummy-4 (13356):
error walking up to an element with a valid windowHandle
Traceback (most recent call last):
File "UIAHandler__init.pyc", line 1338, in getNearestWindowHandle
_ctypes.COMError: (-2147467259, 'Unspecified error', (None, None, None, 0, None))
DEBUGWARNING - NVDAObjects.call (19:02:28.030) - Dummy-4 (13356):
Invalid NVDAObject: no windowHandle
Traceback (most recent call last):
File "NVDAObjects__init.pyc", line 96, in call
File "NVDAObjects\UIA\init.pyc", line 1553, in init
NVDAObjects.InvalidNVDAObject: no windowHandle
DEBUG - UIAHandler.UIAHandler.IUIAutomationNotificationEventHandler_HandleNotificationEvent (19:02:28.030) - Dummy-4 (13356):
HandleNotificationEvent: Ignoring because no object: NotificationProcessing=3 displayString=None activityId=Windows.Shell.VoiceInput.StoppedListening
DEBUG - UIAHandler.UIAHandler.IUIAutomationNotificationEventHandler_HandleNotificationEvent (19:02:28.030) - Dummy-4 (13356):
handleNotificationEvent called with notificationKind ActionCompleted, notificationProcessing ImportantAll, displayString Microphone sleep, activityID Windows.Shell.VoiceAccess, for element Microphone sleep button with automationID MicrophoneButtonControl, className Button and frameworkID XAML
DEBUG - NVDAObjects.UIA.UIA.init (19:02:28.030) - Dummy-4 (13356):
No windowHandle for UIA NvDAObject. Searching UIA element ancestry for nearest windowHandle
DEBUG - UIAHandler.UIAHandler.getNearestWindowHandle (19:02:28.030) - Dummy-4 (13356):
Locating nearest ancestor windowHandle for element Microphone sleep button with automationID MicrophoneButtonControl, className Button and frameworkID XAML
DEBUGWARNING - UIAHandler.UIAHandler.getNearestWindowHandle (19:02:28.032) - Dummy-4 (13356):
error walking up to an element with a valid windowHandle
Traceback (most recent call last):
File "UIAHandler\init__.pyc", line 1338, in getNearestWindowHandle
_ctypes.COMError: (-2147467259, 'Unspecified error', (None, None, None, 0, None))
DEBUGWARNING - NVDAObjects.call (19:02:28.032) - Dummy-4 (13356):
Invalid NVDAObject: no windowHandle
Traceback (most recent call last):
File "NVDAObjects\init.pyc", line 96, in call
File "NVDAObjects\UIA\init.pyc", line 1553, in init__
NVDAObjects.InvalidNVDAObject: no windowHandle
DEBUG - UIAHandler.UIAHandler.IUIAutomationNotificationEventHandler_HandleNotificationEvent (19:02:28.032) - Dummy-4 (13356):
HandleNotificationEvent: Ignoring because no object: NotificationProcessing=0 displayString=Microphone sleep activityId=Windows.Shell.VoiceAccess
System configuration
NVDA installed/portable/running from source:
Installed
NVDA version:
alpha-32918,3aafc821
Windows version:
Windows 11 24H2 preview (build 26100.1150)
Name and version of other software in use when reproducing the issue:
Voice Access 24H2 (10.0.26100)
Other information about your system:
Used as a development workstation, subscribed to Windows Insider/release preview channel
Other questions
Does the issue still occur after restarting your computer?
Yes
Have you tried any other versions of NVDA? If so, please report their behaviors.
Reproducible in 2024.1, 2024.2, 2024.3 beta
If NVDA add-ons are disabled, is your problem still occurring?
Yes
Does the issue still occur after you run the COM Registration Fixing Tool in NVDA's tools menu?
Not applicable
Additional context:
I suspect something odd is happening when fetching UIA element window handle while trying to handle notification event form Voice Access (NVDA needs window handle when constructing objects). At least resolving this will allow NVDA to announce things while focused on Voice Access. Getting NVDA to announce Voice Access status from everywhere can be simple as creating an app module to announce notification display string and no more (because focused app is not Voice Access, thus NVDA will ignore notification event by default).
The biggest impact is that NVDA users will be able to use Voice Access more fully, including getting notificatoins about microphone toggle, error alerts, among others. I provided primitive support for Voice Access (at least letting NVDA announce microphone toggle from the keyboard) via an add-on, but resolving the root issue (UIA notification event handling) would be a more effective solution.
Background:
Windows 11 22H2 (2022 Update/build 22621) introduces Voice Access, a tool to let people dictate text and perform commands using their voices. While Voice Access is active, an overlay window is displayed with controls such as dictated text, microphone state toggle button, and a help button.
Voice Access uses UIA notification event to notify assistive technologies about changes such as microphone state and dictated text. At the moment NVDA does not announce these notifications because it cannot locate the object in question as seen in the log fragment below. Interestingly, NVDA can report notifications if Narrator is also active.
Steps to reproduce:
Prerequisities: Windows 11 22H2 or later on a computer with a microphone.
Actual behavior:
The following assumes Voice Access window is focused:
### Expected behavior: NVDA announces Voice Access notifications while focused on the Voice Access window without Narrator running. The issue of NVDA not announcing UIA notifications from places other than Voice Access can be dealt with as it has to do with focused app check in default UIA notification event handler in NVDA.
NVDA logs, crash dumps and other attachments:
Context: microphone was toggled from Voice Access by pressing Enter.
Input: kb(laptop):enter DEBUG - UIAHandler.UIAHandler.IUIAutomationNotificationEventHandler_HandleNotificationEvent (19:02:28.029) - Dummy-4 (13356): handleNotificationEvent called with notificationKind ActionCompleted, notificationProcessing MostRecent, displayString None, activityID Windows.Shell.VoiceInput.StoppedListening, for element Loading... text with automationID SpeechTextBlock, className TextBlock and frameworkID XAML DEBUG - NVDAObjects.UIA.UIA.init (19:02:28.029) - Dummy-4 (13356): No windowHandle for UIA NvDAObject. Searching UIA element ancestry for nearest windowHandle DEBUG - UIAHandler.UIAHandler.getNearestWindowHandle (19:02:28.029) - Dummy-4 (13356): Locating nearest ancestor windowHandle for element Loading... text with automationID SpeechTextBlock, className TextBlock and frameworkID XAML DEBUGWARNING - UIAHandler.UIAHandler.getNearestWindowHandle (19:02:28.030) - Dummy-4 (13356): error walking up to an element with a valid windowHandle Traceback (most recent call last): File "UIAHandler__init.pyc", line 1338, in getNearestWindowHandle _ctypes.COMError: (-2147467259, 'Unspecified error', (None, None, None, 0, None)) DEBUGWARNING - NVDAObjects.call (19:02:28.030) - Dummy-4 (13356): Invalid NVDAObject: no windowHandle Traceback (most recent call last): File "NVDAObjects__init.pyc", line 96, in call File "NVDAObjects\UIA\init.pyc", line 1553, in init NVDAObjects.InvalidNVDAObject: no windowHandle DEBUG - UIAHandler.UIAHandler.IUIAutomationNotificationEventHandler_HandleNotificationEvent (19:02:28.030) - Dummy-4 (13356): HandleNotificationEvent: Ignoring because no object: NotificationProcessing=3 displayString=None activityId=Windows.Shell.VoiceInput.StoppedListening DEBUG - UIAHandler.UIAHandler.IUIAutomationNotificationEventHandler_HandleNotificationEvent (19:02:28.030) - Dummy-4 (13356): handleNotificationEvent called with notificationKind ActionCompleted, notificationProcessing ImportantAll, displayString Microphone sleep, activityID Windows.Shell.VoiceAccess, for element Microphone sleep button with automationID MicrophoneButtonControl, className Button and frameworkID XAML DEBUG - NVDAObjects.UIA.UIA.init (19:02:28.030) - Dummy-4 (13356): No windowHandle for UIA NvDAObject. Searching UIA element ancestry for nearest windowHandle DEBUG - UIAHandler.UIAHandler.getNearestWindowHandle (19:02:28.030) - Dummy-4 (13356): Locating nearest ancestor windowHandle for element Microphone sleep button with automationID MicrophoneButtonControl, className Button and frameworkID XAML DEBUGWARNING - UIAHandler.UIAHandler.getNearestWindowHandle (19:02:28.032) - Dummy-4 (13356): error walking up to an element with a valid windowHandle Traceback (most recent call last): File "UIAHandler\init__.pyc", line 1338, in getNearestWindowHandle _ctypes.COMError: (-2147467259, 'Unspecified error', (None, None, None, 0, None)) DEBUGWARNING - NVDAObjects.call (19:02:28.032) - Dummy-4 (13356): Invalid NVDAObject: no windowHandle Traceback (most recent call last): File "NVDAObjects\init.pyc", line 96, in call File "NVDAObjects\UIA\init.pyc", line 1553, in init__ NVDAObjects.InvalidNVDAObject: no windowHandle DEBUG - UIAHandler.UIAHandler.IUIAutomationNotificationEventHandler_HandleNotificationEvent (19:02:28.032) - Dummy-4 (13356): HandleNotificationEvent: Ignoring because no object: NotificationProcessing=0 displayString=Microphone sleep activityId=Windows.Shell.VoiceAccess
System configuration
NVDA installed/portable/running from source:
Installed
NVDA version:
alpha-32918,3aafc821
Windows version:
Windows 11 24H2 preview (build 26100.1150)
Name and version of other software in use when reproducing the issue:
Voice Access 24H2 (10.0.26100)
Other information about your system:
Used as a development workstation, subscribed to Windows Insider/release preview channel
Other questions
Does the issue still occur after restarting your computer?
Yes
Have you tried any other versions of NVDA? If so, please report their behaviors.
Reproducible in 2024.1, 2024.2, 2024.3 beta
If NVDA add-ons are disabled, is your problem still occurring?
Yes
Does the issue still occur after you run the COM Registration Fixing Tool in NVDA's tools menu?
Not applicable
Additional context:
I suspect something odd is happening when fetching UIA element window handle while trying to handle notification event form Voice Access (NVDA needs window handle when constructing objects). At least resolving this will allow NVDA to announce things while focused on Voice Access. Getting NVDA to announce Voice Access status from everywhere can be simple as creating an app module to announce notification display string and no more (because focused app is not Voice Access, thus NVDA will ignore notification event by default).
The biggest impact is that NVDA users will be able to use Voice Access more fully, including getting notificatoins about microphone toggle, error alerts, among others. I provided primitive support for Voice Access (at least letting NVDA announce microphone toggle from the keyboard) via an add-on, but resolving the root issue (UIA notification event handling) would be a more effective solution.
Thanks.