Enhanced support for form controls in MS Word

LeonarddeR commented 7 years ago

This ticket is related to #2295, #4709 and #6643.

Example materials:

form.docx. This file contains an edit box and a combo box created using html, along with labels and preset values. It seems they have been imported as legacy controls. NVDA behaves very strange and inconsistent in this document. Behaviour I've seen so far:
- NVDA just says embedded object
- NVDA says embedded object edit value
- NVDA says embedded object edit value and repeats itself multiple times
New and legacy form controls.docx. This document has been made using Word itself, you will need to enable the developer tab for this. During the creation of this document, I noticed that NVDA started being sluggish after I inserted the legacy form controls. Those legacy controls also cause the strange behaviour described above.

Overall, it is not possible to change values in form controls in a reliable way. I think this is a show stopper for #6643. Especially the sluggishness with legacy controls is something which should also be fixed before we even start thinking about making editing them accessible.

LeonarddeR commented 7 years ago

cc @derekriemer, @michaeldcurran, @josephsl, @Qchristensen

LeonarddeR commented 6 years ago

It seems that #7849 doesn't improve this much. The new style check boxes are navigable using browse mode, they can't be toggled in browse mode though. All other form controls do not have proper control types assigned.

Adriani90 commented 5 months ago

Status with NVDA alpha-32185,7e31f30f (2024.3.0.32185) in MS Word 365, both UIA enabled and disabled, behavior is the same:

First document with HTML based edit field and combo box:

When focusing the embeded object, You have to press shift+right arrow to expand it and the caret or the focus lands in the edit field or in the combo box.

In the edit field, NVDA does not report anything unless you change the window back and forth with alt+tab. When UIA is enabled, following is written to the log:

IO - inputCore.InputManager.executeGesture (23:31:35.676) - winInputHook (24896):
Input: kb(laptop):shift+rightArrow
DEBUG - UIAHandler.shouldUseUIAInMSWord (23:31:35.691) - MainThread (17536):
Using UIA due to suitable Office version: (16, 0, 17531)
IO - speech.speech.speak (23:31:35.729) - MainThread (17536):
Speaking ['Eingabefeld', CancellableSpeech (still valid), 'Leer']
DEBUGWARNING - NVDAObjects.IAccessible.IAccessible._getIA2RelationFirstTarget (23:31:35.736) - MainThread (17536):
Unable to use _getIA2TargetsForRelationsOfType, fallback to _IA2Relations.
DEBUG - NVDAObjects.IAccessible.IAccessible._get__IA2Relations (23:31:35.736) - MainThread (17536):
Not an IA2.IAccessible2
DEBUG - NVDAObjects.IAccessible.IAccessible._getIA2RelationFirstTarget (23:31:35.736) - MainThread (17536):
Unable to fetch _IA2Relations
Traceback (most recent call last):
File "NVDAObjects\IAccessible\__init__.pyc", line 1635, in _getIA2RelationFirstTarget
File "baseObject.pyc", line 62, in __get__
File "baseObject.pyc", line 168, in _getPropertyViaCache
File "NVDAObjects\IAccessible\__init__.pyc", line 1550, in _get__IA2Relations
NotImplementedError
IO - speech.speech.speak (23:31:35.776) - MainThread (17536):
Speaking ['Eingabefeld', CancellableSpeech (still valid)]
IO - inputCore.InputManager.executeGesture (23:31:36.476) - winInputHook (24896):
Input: kb(laptop):leftArrow
IO - inputCore.InputManager.executeGesture (23:31:36.826) - winInputHook (24896):
Input: kb(laptop):rightArrow
IO - inputCore.InputManager.executeGesture (23:31:36.996) - winInputHook (24896):
Input: kb(laptop):downArrow
IO - inputCore.InputManager.executeGesture (23:31:37.252) - winInputHook (24896):
Input: kb(laptop):upArrow
IO - inputCore.InputManager.executeGesture (23:31:37.466) - winInputHook (24896):
Input: kb(laptop):downArrow
IO - inputCore.InputManager.executeGesture (23:31:37.696) - winInputHook (24896):
Input: kb(laptop):upArrow
IO - inputCore.InputManager.executeGesture (23:31:38.158) - winInputHook (24896):
Input: kb(laptop):rightArrow
IO - inputCore.InputManager.executeGesture (23:31:38.441) - winInputHook (24896):
Input: kb(laptop):a
DEBUG - UIAHandler.shouldUseUIAInMSWord (23:31:38.476) - MainThread (17536):
Using UIA due to suitable Office version: (16, 0, 17531)
IO - inputCore.InputManager.executeGesture (23:31:38.656) - winInputHook (24896):
Input: kb(laptop):a
IO - inputCore.InputManager.executeGesture (23:31:38.881) - winInputHook (24896):
Input: kb(laptop):leftArrow
IO - inputCore.InputManager.executeGesture (23:31:39.126) - winInputHook (24896):
Input: kb(laptop):leftArrow

On the combo box, NVDA reports it role, but when trying to change values aNVDA is silent. NVDA gives an error on every arrow key press, and Following is written to the log:

IO - inputCore.InputManager.executeGesture (23:18:59.368) - winInputHook (2544):
Input: kb(laptop):downArrow
DEBUG - editableText.EditableText._hasCaretMoved (23:18:59.386) - MainThread (28596):
Caret move detected using event. Elapsed 0 sec, retries 0
IO - speech.speech.speak (23:18:59.464) - MainThread (28596):
Speaking ['Eingebettetes Objekt']
IO - inputCore.InputManager.executeGesture (23:19:00.265) - winInputHook (2544):
Input: kb(laptop):shift+rightArrow
DEBUG - UIAHandler.shouldUseUIAInMSWord (23:19:00.285) - Dummy-1 (10952):
User does not want UIA in MS Word unless necessary
IO - speech.speech.speak (23:19:00.295) - MainThread (28596):
Speaking ['\x01 ausgewählt']
IO - speech.speech.speak (23:19:00.346) - MainThread (28596):
Speaking ['Kombinationsfeld', 'value 3', CancellableSpeech (still valid)]
IO - speech.speech.speak (23:19:00.366) - MainThread (28596):
Speaking ['Kombinationsfeld', 'value 3', CancellableSpeech (still valid)]
IO - inputCore.InputManager.executeGesture (23:19:01.976) - winInputHook (2544):
Input: kb(laptop):alt+downArrow
IO - inputCore.InputManager.executeGesture (23:19:02.356) - winInputHook (2544):
Input: kb(laptop):downArrow
IO - inputCore.InputManager.executeGesture (23:19:02.626) - winInputHook (2544):
Input: kb(laptop):upArrow
DEBUG - UIAHandler.shouldUseUIAInMSWord (23:19:02.658) - MainThread (28596):
User does not want UIA in MS Word unless necessary
IO - inputCore.InputManager.executeGesture (23:19:02.946) - winInputHook (2544):
Input: kb(laptop):downArrow
IO - inputCore.InputManager.executeGesture (23:19:03.126) - winInputHook (2544):
Input: kb(laptop):downArrow
DEBUG - UIAHandler.shouldUseUIAInMSWord (23:19:03.248) - Dummy-1 (10952):
User does not want UIA in MS Word unless necessary
IO - inputCore.InputManager.executeGesture (23:19:03.366) - winInputHook (2544):
Input: kb(laptop):upArrow
IO - inputCore.InputManager.executeGesture (23:19:03.656) - winInputHook (2544):
Input: kb(laptop):downArrow
DEBUG - UIAHandler.shouldUseUIAInMSWord (23:19:03.796) - Dummy-1 (10952):
User does not want UIA in MS Word unless necessary
IO - inputCore.InputManager.executeGesture (23:19:03.896) - winInputHook (2544):
Input: kb(laptop):downArrow
IO - inputCore.InputManager.executeGesture (23:19:04.166) - winInputHook (2544):
Input: kb(laptop):upArrow
DEBUG - UIAHandler.shouldUseUIAInMSWord (23:19:04.300) - Dummy-1 (10952):
User does not want UIA in MS Word unless necessary
IO - inputCore.InputManager.executeGesture (23:19:04.336) - winInputHook (2544):
Input: kb(laptop):upArrow
IO - inputCore.InputManager.executeGesture (23:19:04.506) - winInputHook (2544):
Input: kb(laptop):upArrow
IO - inputCore.InputManager.executeGesture (23:19:05.016) - winInputHook (2544):
Input: kb(laptop):enter
IO - inputCore.InputManager.executeGesture (23:19:05.476) - winInputHook (2544):
Input: kb(laptop):upArrow
IO - inputCore.InputManager.executeGesture (23:19:05.826) - winInputHook (2544):
Input: kb(laptop):downArrow
DEBUG - UIAHandler.shouldUseUIAInMSWord (23:19:05.855) - MainThread (28596):
User does not want UIA in MS Word unless necessary
IO - inputCore.InputManager.executeGesture (23:19:06.218) - winInputHook (2544):
Input: kb(laptop):upArrow
DEBUG - UIAHandler.shouldUseUIAInMSWord (23:19:06.522) - Dummy-1 (10952):
User does not want UIA in MS Word unless necessary
IO - inputCore.InputManager.executeGesture (23:19:07.036) - winInputHook (2544):
Input: kb(laptop):alt+tab
IO - speech.speech.speak (23:19:07.295) - MainThread (28596):
Speaking ['Downloads']
IO - speech.speech.speak (23:19:07.305) - MainThread (28596):
Speaking ['Elementansicht', 'Liste']
IO - speech.speech.speak (23:19:07.315) - MainThread (28596):
Speaking ['Heute', 'Gruppierung', 'ausgeklappt']
IO - speech.speech.speak (23:19:07.315) - MainThread (28596):
Speaking ['form.docx', '1 von 68']
IO - inputCore.InputManager.executeGesture (23:19:07.633) - winInputHook (2544):
Input: kb(laptop):alt+tab
IO - speech.speech.speak (23:19:07.715) - MainThread (28596):
Speaking ['form.docx - Word', '2 von 6']
DEBUGWARNING - IAccessibleHandler.internalWinEventHandler._shouldGetEvents (23:19:07.836) - MainThread (28596):
Foreground took too long to change. Foreground still 526986 (OpusApp). Should be 657032 (XamlExplorerHostIslandWindow)
DEBUG - UIAHandler.shouldUseUIAInMSWord (23:19:07.863) - MainThread (28596):
User does not want UIA in MS Word unless necessary
IO - speech.speech.speak (23:19:07.906) - MainThread (28596):
Speaking ['form.docx - Word', CancellableSpeech (still valid)]
IO - speech.speech.speak (23:19:07.906) - MainThread (28596):
Speaking ['Kombinationsfeld', 'value 1', CancellableSpeech (still valid)]
ERROR - eventHandler.executeEvent (23:19:07.906) - MainThread (28596):
error executing event: caret on <NVDAObjects.Dynamic_IAccessibleDisplayModelEditableTextWindowNVDAObject object at 0x05739F70> with extra args of {}
Traceback (most recent call last):
File "eventHandler.pyc", line 345, in executeEvent
File "eventHandler.pyc", line 116, in __init__
File "eventHandler.pyc", line 125, in next
File "NVDAObjects\IAccessible\__init__.pyc", line 1769, in event_caret
File "NVDAObjects\__init__.pyc", line 1366, in event_caret
File "review.pyc", line 183, in handleCaretMove
File "documentBase.pyc", line 74, in makeTextInfo
File "displayModel.pyc", line 298, in __init__
File "textInfos\offsets.pyc", line 478, in __init__
File "displayModel.pyc", line 619, in _getCaretOffset
AttributeError: 'NoneType' object has no attribute 'toLTRB'
ERROR - eventHandler.executeEvent (23:19:07.916) - MainThread (28596):
error executing event: caret on <NVDAObjects.Dynamic_IAccessibleDisplayModelEditableTextWindowNVDAObject object at 0x05739F70> with extra args of {}
Traceback (most recent call last):
File "eventHandler.pyc", line 345, in executeEvent
File "eventHandler.pyc", line 116, in __init__
File "eventHandler.pyc", line 125, in next
File "NVDAObjects\IAccessible\__init__.pyc", line 1769, in event_caret
File "NVDAObjects\__init__.pyc", line 1366, in event_caret
File "review.pyc", line 183, in handleCaretMove
File "documentBase.pyc", line 74, in makeTextInfo
File "displayModel.pyc", line 298, in __init__
File "textInfos\offsets.pyc", line 478, in __init__
File "displayModel.pyc", line 619, in _getCaretOffset
AttributeError: 'NoneType' object has no attribute 'toLTRB'

When UIA in MS Word is enabled, on pressing shift+right arrow to expand the embeded object, following error is written to the log:

DEBUGWARNING - NVDAObjects.IAccessible.IAccessible._getIA2RelationFirstTarget (23:28:19.480) - MainThread (28596):
Unable to use _getIA2TargetsForRelationsOfType, fallback to _IA2Relations.
DEBUG - NVDAObjects.IAccessible.IAccessible._get__IA2Relations (23:28:19.480) - MainThread (28596):
Not an IA2.IAccessible2
DEBUG - NVDAObjects.IAccessible.IAccessible._getIA2RelationFirstTarget (23:28:19.480) - MainThread (28596):
Unable to fetch _IA2Relations
Traceback (most recent call last):
  File "NVDAObjects\IAccessible\__init__.pyc", line 1635, in _getIA2RelationFirstTarget
  File "baseObject.pyc", line 62, in __get__
  File "baseObject.pyc", line 168, in _getPropertyViaCache
  File "NVDAObjects\IAccessible\__init__.pyc", line 1550, in _get__IA2Relations
NotImplementedError

Adriani90 commented 5 months ago

The developer info for the edit field is as follows:

INFO - globalCommands.script_navigatorObject_devInfo (23:33:24.946) - MainThread (17536):
Developer info for navigator object:
name: None
role: EDITABLETEXT
processID: 25184
roleText: None
states: FOCUSABLE, FOCUSED
isFocusable: True
hasFocus: True
Python object: <NVDAObjects.IAccessible.IAccessible object at 0x01230170>
Python class mro: (<class 'NVDAObjects.IAccessible.IAccessible'>, <class 'NVDAObjects.window.Window'>, <class 'NVDAObjects.NVDAObject'>, <class 'documentBase.TextContainerObject'>, <class 'baseObject.ScriptableObject'>, <class 'baseObject.AutoPropertyObject'>, <class 'garbageHandler.TrackedObject'>, <class 'object'>)
description: None
location: RectLTWH(left=496, top=315, width=107, height=36)
value: None
TextInfo: <class 'NVDAObjects.NVDAObjectTextInfo'>
appModule: AppModule(winword, appName='winword', processID=25184)
appModule.productName: 'Microsoft Office'
appModule.productVersion: '16.0.17531.20152'
appModule.helperLocalBindingHandle: c_long(73477104)
windowHandle: 9636202
windowClassName: 'F3 Server 9e980000'
windowControlID: 0
windowStyle: 1442840576
extendedWindowStyle: 0
windowThreadID: 20816
windowText: ''
displayText: ''
IAccessibleObject: <POINTER(IAccessible) ptr=0x6897e48 at 9d66b20>
IAccessibleChildID: 0
IAccessible event parameters: windowHandle=9636202, objectID=9, childID=0
IAccessible accName: exception: (-2147467263, 'Nicht implementiert', (None, None, None, 0, None))
IAccessible accRole: ROLE_SYSTEM_TEXT
IAccessible accState: STATE_SYSTEM_FOCUSED, STATE_SYSTEM_FOCUSABLE, STATE_SYSTEM_VALID (1048580)
IAccessible accDescription: None
IAccessible accValue: None

The developer info for the combo box is as follows:

INFO - globalCommands.script_navigatorObject_devInfo (23:37:39.349) - MainThread (17536):
Developer info for navigator object:
name: None
role: COMBOBOX
processID: 25184
roleText: None
states: FOCUSABLE, FOCUSED
isFocusable: True
hasFocus: True
Python object: <NVDAObjects.Dynamic_IAccessibleDisplayModelEditableTextWindowNVDAObject object at 0x09DD90D0>
Python class mro: (<class 'NVDAObjects.Dynamic_IAccessibleDisplayModelEditableTextWindowNVDAObject'>, <class 'NVDAObjects.IAccessible.IAccessible'>, <class 'NVDAObjects.window.DisplayModelEditableText'>, <class 'NVDAObjects.behaviors.EditableTextWithoutAutoSelectDetection'>, <class 'editableText.EditableTextWithoutAutoSelectDetection'>, <class 'NVDAObjects.behaviors.EditableText'>, <class 'NVDAObjects.behaviors.EditableTextWithSuggestions'>, <class 'NVDAObjects.behaviors.InputFieldWithSuggestions'>, <class 'NVDAObjects.behaviors.EditableTextBase'>, <class 'editableText.EditableText'>, <class 'NVDAObjects.window.Window'>, <class 'NVDAObjects.NVDAObject'>, <class 'documentBase.TextContainerObject'>, <class 'baseObject.ScriptableObject'>, <class 'baseObject.AutoPropertyObject'>, <class 'garbageHandler.TrackedObject'>, <class 'object'>)
description: None
location: None
value: 'value 2'
TextInfo: <class 'displayModel.EditableTextDisplayModelTextInfo'>
appModule: AppModule(winword, appName='winword', processID=25184)
appModule.productName: 'Microsoft Office'
appModule.productVersion: '16.0.17531.20152'
appModule.helperLocalBindingHandle: c_long(73477104)
windowHandle: 526936
windowClassName: 'F3 Server 9e980000'
windowControlID: 0
windowStyle: 1442840576
extendedWindowStyle: 0
windowThreadID: 20816
windowText: ''
displayText: exception: cannot unpack non-iterable NoneType object
IAccessibleObject: <POINTER(IAccessible) ptr=0x6926390 at 9da1c10>
IAccessibleChildID: 0
IAccessible event parameters: windowHandle=526936, objectID=-4, childID=0
IAccessible accName: exception: (-2147467263, 'Nicht implementiert', (None, None, None, 0, None))
IAccessible accRole: ROLE_SYSTEM_COMBOBOX
IAccessible accState: STATE_SYSTEM_FOCUSED, STATE_SYSTEM_FOCUSABLE, STATE_SYSTEM_VALID (1048580)
IAccessible accDescription: None
IAccessible accValue: 'value 2'

Adriani90 commented 5 months ago

New and legacy form fields: Enabling UIA makes this form quite inaccessible, so I tested with UIA disabled.
- The HTML based form fields are actually the same like legacy form fields. In the second document, the legacy form fields can be expanded by pressing shift+right arrow, and the same behavior like in the tests above applies.

For new form fields though, NVDA seem to report the labels properly if there is a help text. Edit boxes and checkboxes can be navigated by tab if you focus one of the controls with arrow keys, but you cannot navigate them by tab when you focus its label. In that case NVDA will add tab characters next to the system caret.
Check boxes added via new form fields can be enabled and disabled properly, but NVDA doesn't announce the state change when pressing spacebar. This also applies for legacy / HTML based checkboxes.
Adding combo boxes via new form fields, NVDA can only expand them using alt+down arrow when using browse mode.

nvaccess / nvda

Enhanced support for form controls in MS Word #6944