Qt 5.11+ accessibility requires updates on the NVDA side

[ghost]

Qt 5.11 replaced its legacy MSAA-based Windows accessibility backend with a new implementation based on UI Automation. This seems to be causing some issues with NVDA. I'm currently investigating the following bug report, which includes a detailed description of the issues, as well as a sample project to reproduce them:

https://bugreports.qt.io/browse/QTBUG-69472

While some of the issues may result from bugs in our implementation, some others may require updates in the NVDA side, which uses a custom app module for Qt applications, which may no longer apply since Qt 5.11.

I took a look at NVDA sources and saw that it supports MSAA and UI Automation, and has app modules that add custom behavior and fixes for many applications and frameworks, including Qt. The Qt app module in NVDA is based on MSAA/IAccessible, used in previous Qt versions. When used with a Qt 5.11 application, NVDA is recognizing that it supports UI Automation and using its UI Automation module.

However, this module seems to be lacking some functionality. For instance, the automatic detection of changes in text selection seems to have been disabled for some reason. The Qt framework is raising "UIA_Text_TextSelectionChangedEventId" notifications to inform UIA clients of changes in selection. However, NVDA seems to be purposely ignoring it (the mapping from UIA_Text_TextSelectionChangedEventId to the "caret" event is commented out in the _UIAHandler.py source). Also, NVDA's UIA module uses the "EditableTextWithoutAutoSelectDetection" class, instead of the "EditableTextWithAutoSelectDetection" used on the IAccessible module. I tried enabling the UIA_Text_TextSelectionChangedEventId mapping and changing the UIA module to use EditableTextWithAutoSelectDetection, and some of the compatibility issues reported, like NVDA failing to speak the mouse selection, or the lack of updates in the Braille display where fixed. However, I'm not familiar with the NVDA code, and that change will likely break other stuff.

I would suggest keeping the current custom app module in place for supporting Qt versions prior to 5.11 and using the UIA module for >= 5.11 (i.e. whenever UIA support is detected), while enabling the necessary features to make it work properly. I can help with changes on the Qt side.

Also, please instead of creating a new custom app module for Qt 5.11+ to work around bugs in our implementation or behavior unexpected for a proper UI Automation server, please report them to us so we can have them fixed on our side instead (you can assign bug reports to me).

Note: the new accessibility backend is a pure UI Automation implementation. Legacy MSAA clients are supported through the UIA-to-MSAA bridge provided by the MS UIA framework in the middle. However, the use of the bridge was broken in the initial 5.11 release, but will be fixed in 5.11.2 (the fix is already in our 5.11 branch, if you build Qt yourself). However, the use through the bridge does not provide all functionality that may be required. In particular, it does not provide IAccessible2 support, only IAccessible. Some features like caret movement detection are in IAccessible2, so I think a proper UI Automation interfacing is the way to go, instead of trying to use it in MSAA mode (also, even going through this legacy mode would also require changes in NVDA, as it detects the UI Automation support in Qt 5.11 and seems to ignore the MSAA-based app module).

[josephsl]

CC @MichaelDCurran and @JCSTeh

[LeonarddeR]

@anrocqt commented on 7 Aug 2018, 16:42 CEST:

... as well as a sample project to reproduce them:

I might be missing something, but could you please provide a executable binary of this application? That will make testing a whole lot easier I think.

[ghost]

SimpleBox.zip

@leonardder: Added a binary of the sample application statically built against the current code in our 5.11 branch.

[LeonarddeR]

@anrocqt commented on 7 Aug 2018, 16:42 CEST:

... which uses a custom app module for Qt applications, which may no longer apply since Qt 5.11.

Strictly spoken, NVDA uses special NVDAObject types for some QT widgets that recide in the NVDAObjects.IAccessible package. These will not apply to the UIA implementation at all, as UIA normally has preference over MSAA/IAccessible.

For instance, the automatic detection of changes in text selection seems to have been disabled for some reason. The Qt framework is raising "UIA_Text_TextSelectionChangedEventId" notifications to inform UIA clients of changes in selection. However, NVDA seems to be purposely ignoring it (the mapping from UIA_Text_TextSelectionChangedEventId to the "caret" event is commented out in the _UIAHandler.py source).

@JCSTeh: Could you elaborate on why you commented this out, according to git blame?

As a side note, I noticed that NVDA is unable to detect bold and itallic text in the text browser. @anrocqt: could this implementation be incomplete on the QT side?

[jcsteh]

These events were disabled because certain UIA providers (I think WPF 2.5 or something like that from memory) flood these events and cause NVDA to become unusable. There was an issue filed for this, but I don't know the number offhand. Because there's no way we can currently selectively blacklist certain events in specific apps at the UIA core level, we had to disable them altogether. The next release of Windows 10 includes UIA core changes to deal with bad providers like this, but that isn't going to help earlier versions of Windows.

[derekriemer]

just want to make sure this doesn't stall, because QT is hugely important.

[LeonarddeR]

@anrocqt: Testing with JAWS 2018 revealed that the formatting changes are not detected within the text browser. The font size is reported as 0 and the bold and italic states are not reported.

@jcsteh commented on 10 aug. 2018 23:12 CEST:

Because there's no way we can currently selectively blacklist certain events in specific apps at the UIA core level, we had to disable them altogether.

Do you recall whether these freezes in NVDA were occurring just because of the flood of text selection events, or because every text selection event executed an event_caret internally?

[LeonarddeR]

@anrocqt: Is there anything to add now QT 5.12 has seen the light?

[shoogle]

Can this be revisited now? This has affected MuseScore and other Qt applications. It seems that Qt is doing everything right, but NVDA is purposefully ignoring accessibility events to compensate for bugs in other (non-Qt) applications.

Here are the relevant lines in nvda/_UIAHandler.py where various events are being ignored (they are commented out).

@jcsteh

These events were disabled because certain UIA providers (I think WPF 2.5 or something like that from memory) flood these events and cause NVDA to become unusable.

I appreciate that the changes allowed NVDA to work with those applications, but it also removed any incentive for developers to fix bugs in those applications. Worse, if the developers test with NVDA they might come to the conclusion that there are no bugs in their application.

In the past it was necessary for screen readers to go to great lengths to make non-accessible applications accessible. However, times are changing, and the onus is now on application developers to make their applications accessible. Let's reward applications that get it right rather than penalising them for the sake of preserving accessibility in applications that get it wrong.

[jcsteh]

It's not just about allowing NVDA to work with those applications. The problem is that with these events enabled, any app based on those frameworks (whether the author cares about accessibility or not) will cause a complete hang of NVDA. Prior to Windows 10, there is no way NVDA can work around this. That said, this should only affect reporting of text selection via the mouse. Text selection via the keyboard should work, as should anything else (focus, property changes, etc.).

I don't understand how this could cause the issues being experienced with MuseScore. Surely it's not using text selection events to report cursoring around the Score?

[LeonarddeR]

I've compared NVDA with Narrator with regard to MuseScore, and at least with NVDA, combo boxes do not behave as they should, that is, arrowing through the combo box without opening it does not announce selection changes, whereas with Narrator, this works just fine.

[LeonarddeR]

The QComboBox comboboxes consist of a root object (the combobox itself) and its children (a list with the combobox items, and an edit control). It looks like that NVDA is not receiving any PropertyChangedEvents when changing the value of the combobox without expanding it. Note that eventHandler.shouldAcceptEvent tends to drop some events, but that's certainly not the case here, as the events aren't even entering the event handler of the UIAHandler. Therefore, I have no idea what's going on here. Cc @michaelDCurran

Having said that, the QComboBox neither implements a ExpandCollapse nor a Selection pattern, which, according to the Microsoft Docs, are required. Cc @anrocqt

[shoogle]

As @leonardder said, navigation within unexpanded combo boxes is not being announced by NVDA. Interestingly, this was working under Qt 5.9 (which used MSAA) but not under Qt 5.12 (which uses UI Automation). There are other examples of this:

• Under Qt 5.12 only control names (i.e. not descriptions) are announced by NVDA. Both names and descriptions were announced under 5.9.
• Changes to selections made in the Score View were announced under Qt 5.9 and not under Qt 5.12.

If you want to confirm that the above is indeed true then you will need an old version of MuseScore that uses Qt 5.9. Subsequent versions use Qt 5.12, and the MuseScore developers are reluctant to switch back.

Regarding the Score View, I suspect that MuseScore was using a "trick" to get NVDA to read whatever was in the status bar, so it's possible that this is an issue at MuseScore's end. Nevertheless, the "trick" was working under Qt 5.9 and MSAA and now it is broken under Qt 5.12 and UI Automation.

• Here is the code that notifies that the text has changed.
  • We send QAccessibleValueChangeEvent with the object being the score view and the value being the new status text.
• According to Qt source code, QAccessibleValueChangeEvent raises UIA_RangeValueValuePropertyId.

If we are doing something wrong here then it would be very helpful if somebody could point us in the direction of a tutorial / example of how to implement accessibility for a custom control. Most accessibility guides just say "don't use custom controls, use built-in controls", but MuseScore's Score View is a custom control out of necessity (after all, if there was a standard way to display musical symbols in Windows then there would be no reason for MuseScore to exist).

[shoogle]

It would be great if somebody could try building it with the relevant lines in nvda/_UIAHandler.py uncommented so that all events are handled, and then testing it with MuseScore to see which of the issues highlighted so far still remain, if any:

1. Arrowing through non-expanded combo boxes is unannounced.
2. Control descriptions are unannounced.
3. Navigation and selection changes within the Score View is unannounced.

I tried to build NVDA myself but I don't have enough room on my PC for all the dependencies.

[LeonarddeR]

I've already done this, without luck.

[shoogle]

@leonardder, thanks! You didn't notice any change at all?

@jcsteh

Surely [MuseScore]'s not using text selection events to report cursoring around the Score?

Cursoring changes the selection, but what event would you expect to see here?

[jcsteh]

I'm not sure. It really depends how the score accessibility is implemented and I've never tested it. What I do know is that it's not an editable text control (and text infers very specific ideas about characters, words and lines), so text selection events definitely aren't appropriate.

[shoogle]

Is there a UI Automation event we can fire that would force NVDA to read the text in the status bar, or arbitrary text of our choosing that is not necessarily displayed anywhere on the screen?

[LeonarddeR]

@shoogle: If I understand you correctly, all was well with older versions of MuseScore based on qt<5.11. What was the latest version that utilized this, so I can investigate the differences?

[jcsteh]

I haven't investigated or tested this and don't have time to do so. The following is provided in the hopes that it may help, but without any guarantee of usefulness. :)

  * We send [`QAccessibleValueChangeEvent`](https://doc.qt.io/qt-5/qaccessiblevaluechangeevent.html) with the [object being the score view](https://github.com/musescore/MuseScore/blob/dd3e6859565702bac7905ab424d3314ff82245d5/mscore/scoreaccessibility.cpp#L194) and the [value being the new status text](https://github.com/musescore/MuseScore/blob/dd3e6859565702bac7905ab424d3314ff82245d5/mscore/scoreaccessibility.cpp#L177-L179).

* According to [Qt source code](https://github.com/qt/qtbase/blob/5733dfbd90fd059e7310786faefb022b00289592/src/plugins/platforms/windows/uiautomation/qwindowsuiamainprovider.cpp#L155), `QAccessibleValueChangeEvent` raises `UIA_RangeValueValuePropertyId`.

The way MSAA and UIA handle value is quite different. In MSAA, there is only one value change event and it represents a change to the accValue property, which is generally a string. In UIA, there are two concepts of value, represented by the RangeValue pattern (numeric) and Value pattern (text). It looks like Qt only fires an event for the RangeValue change, but I'm guessing that the value in this case is text.

To make things more confusing, NVDA only has one concept of value. If the RangeValue pattern is present, it will choose RangeValue. Otherwise, it chooses Value. Qt exposes the RangeValue pattern if the accessible has the value interface (numeric). I don't know whether this control exposes that or not; I guess not, since it's not a numeric control.

So, to fix this, I think:

1. This control must not expose the numeric value interface. (It probably doesn't already, but I'm not sure.)
2. In Qt, if the value interface isn't present, it should fire the ValueValue property change event, not the RangeValueValue property change event.

As an ugly, hacky workaround, you could perhaps use the name instead of the value to expose the information. That would of course override any proper name you have set.

[LeonarddeR]

Dropbox 70 switched to qt 5.12 and also suffers from the combobox issue. I'm marking this P2 to start with.

Update: Also discovered that JAWS suffers from exactly the same issue with regard to combo boxes. Selection changes are only exposed when the combobox is expanded. Update2: Also reproduced with Narrator on Windows 10 1809. Will test with 1903 later today.

[ghost]

Hi. Thanks for reporting the issues you found with the Qt accessibility framework on Windows. I apologize for the delayed response. About some of them:

• "It looks like that NVDA is not receiving any PropertyChangedEvents when changing the value of the combo box without expanding it."
• "QComboBox neither implements a ExpandCollapse nor a Selection pattern, which, according to the Microsoft Docs, are required."
• "Changes to selections made in the Score View were announced under Qt 5.9 and not under Qt 5.12."

In spite of what the docs say, it seems Microsoft is not implementing the Selection pattern in their own combo boxes. I checked with the Inspect utility, examining a combo box in the folder options dialog in File Explorer, on Windows 10, and it only implements the Value and ExpandCollapse patterns. And it is working with NVDA. Only the list view inside the combo box structure implements the Selection pattern, as does Qt. Also, the ExpandCollapse pattern makes NVDA say "Expanded"/"Collapsed", but apparently only when doing it with the mouse, which does not seem to be a concern.

However, you are right about the missing UI Automation notifications when the combo box is changed with the keyboard without expanding it. Also, notifications were not being sent for updated string-type values (Value pattern), but only for range-based ones (RangeValue pattern) which probably explains the missing updates in MuseScore.

I created this bug report to track the missing notifications: https://bugreports.qt.io/browse/QTBUG-75066 And submitted the following fix for it (under review): https://codereview.qt-project.org/#/c/258622/

It should fix the combo box issues, and hopefully the Score View selection as well.

• "Under Qt 5.12 only control names (i.e. not descriptions) are announced by NVDA. Both names and descriptions were announced under 5.9"

I have checked with Narrator, and it is saying both the names and descriptions, so it may be an issue in NVDA. Perhaps due to differences on how these fields are returned by MSAA and UI Automation.

If you can build Qt 5.12 yourselves with the proposed patch, please let me know if it actually fixes the issues you are experiencing, as well as any other issues you may have found.

Thanks,

André

[LeonarddeR]

As of dropbox 72 stable, they switched to QT 5.12

[shoogle]

Many thanks to @jcsteh for identifying the issue with Value pattern events, and to @anrocqt for fixing them in Qt! I just tested in Qt 5.12.4 and can confirm that the issues with combo boxes and score view selections in MuseScore have disappeared.

"Under Qt 5.12 only control names (i.e. not descriptions) are announced by NVDA. Both names and descriptions were announced under 5.9"

I have checked with Narrator, and it is saying both the names and descriptions, so it may be an issue in NVDA.

Narrator has now stopped announcing descriptions too, at least by default. However, they are available if you press Narrator+0 (i.e. CapsLock and Zero). Does NVDA have a similar shortcut to announce descriptions? (I looked but couldn't find one.)

[jcsteh]

As far as I'm aware, UIA doesn't have a description property. It has a HelpText property, which from what I can see NVDA maps to description. Unless explicitly disabled (NVDA preferences -> Object presentation -> Report object descriptions), NVDA should report descriptions by default.

What does Qt use for description in UIA? Perhaps it uses ItemStatus, which NVDA indeed doesn't report.

[ghost]

Actually, UIA has a "FullDescription" property, which is where Qt is returning the accessible description text. The Microsoft documentation states:

The FullDescription property exposes a localized string which can contain extended description text for an element. FullDescription can contain a more complete description of an element than may be appropriate for the element Name.

[LeonarddeR]

This property is used for description, but only in Edge.

I recall another issue where someone complained about us mapping HelpText to description.

[shoogle]

Unless explicitly disabled (NVDA preferences -> Object presentation -> Report object descriptions), NVDA should report descriptions by default.

Seems a bit verbose. If that option is disabled, is there a shortcut to report the description of the current focus element?

I recall another issue where someone complained about us mapping HelpText to description.

Sounds like FullDescription is the way to go, but perhaps you could check all of these properties (FullDescription, ItemStatus, HelpText, and maybe ToolTip too if it exists) and report whichever is non-empty?

[jcsteh]

Actually, UIA has a "FullDescription" property, which is where Qt is returning the accessible description text.

Ah, sorry; I missed that one. It got added in Windows 10 1703 (relatively recent in the history of UIA). It certainly didn't exist when NVDA's code for description was written. So yes, we could use that, though it'd probably require some conditional code to avoid breakage on earlier versions of Windows.

This property is used for description, but only in Edge.

I recall another issue where someone complained about us mapping HelpText to description.

I wonder where HelpText is actually used these days? Now that there is an alternative (FullDescription), perhaps we should just use that?

Unless explicitly disabled (NVDA preferences -> Object presentation -> Report object descriptions), NVDA should report descriptions by default.

Seems a bit verbose.

Descriptions are generally useful, and because they're reported after the name, role, etc., a user can choose to ignore them.

If that option is disabled, is there a shortcut to report the description of the current focus element?

Not currently.

Sounds like FullDescription is the way to go, but perhaps you could check all of these properties (FullDescription, ItemStatus, HelpText, and maybe ToolTip too if it exists) and report whichever is non-empty?

That would need to be carefully considered. In particular, ItemStatus tends to include info which is duplicated elsewhere; e.g. item position information.

[dpy013]

Improve mumble accessibility: Upgrade qt 5.12 LTS

[dpy013]

Above is the issue discussion of Improve mumble accessibility

[ethindp]

I doubt any updates on NVDA's side need to be done. NVDA interfaces with QT 5.12 and later just fine. The issue is with Mumble in particular: NVDA does the freeze-lock up problem where it continues functioning, but it does not accept keyboard input (i.e. NVDA commands) any longer nor does it interrupt itself. A test using TeamSpeak 3 (as well as custom applications that I've written in C++ using QT 5.12) do not demonstrate this bug at all, so its definitely on Mumbles end of things. The latest mumble release is using QT 5.6.2. After building the Mumble binaries via MXE on a Ubuntu system this morning, I still get the exact same exhibited behavior as before -- no change. Clearly, Mumble is doing something unusual that NVDA does not expect. I wonder if its because Mumble links everything statically and not dynamically that causes the problem... will need to test that.

[chigkim]

Actually, UIA has a "FullDescription" property, which is where Qt is returning the accessible description text.

Ah, sorry; I missed that one. It got added in Windows 10 1703 (relatively recent in the history of UIA). It certainly didn't exist when NVDA's code for description was written. So yes, we could use that, though it'd probably require some conditional code to avoid breakage on earlier versions of Windows.

@jcsteh, is there any progress on announcing UIA_FullDescriptionPropertyId for description?

[Adriani90]

cc: @feerrenrut

[LeonarddeR]

@anrocqt This got a bit out of my view. How is the current state of QT accessibility? I heard rumours of severe brokenness in version 5.14.

[Adriani90]

Now that QT is going more and more towards version 6.0, is there any status update on this issue?

[DrSooom]

Was this issue solved by NVDA 2020.3 Beta 1 – because it wasn't closed?

[michaelDCurran]

Fixed in pr #10889

[LeonarddeR]

@anrocqt If there are any remaining problems, I'd suggest opening new issues for them separately.

[chigkim]

Since NVDA maps FullDescription to description, it no longer announces HelpText. How do users get the message in HelpText when there are both FullDescription and HelpText? Is there a setting or shortcut to report HelpText?

[LeonarddeR]

@michaelDCurran wrote in https://github.com/nvaccess/nvda/pull/10889#issuecomment-610716033:

Adding a helpText property is okay, though perhaps a little too over-engineered if there are no cases of both of them returning valid info at the same time. Do you think that there are situations in UIA where helpText and fullDescription do give different (and valid) results? Rather than perhaps just using fullDescription but falling back to helpText if empty. If we do implement a helpText property, then we can map IAccessible::accHelp to that as well I guess.

May be we should yet consider splitting description and help text. @chigkim: Could you please open a new issue for this?

[cppqtdev]

Hi Team I am facing one issue with NVDA Screen Reader when the focus goes to the Listview in Qt /QML Application
it is not reading 1 of 5 ,Let's total 5 item in the list and current item is 1 . then it should read name of the item and then read selected and then 1 of 5 or 1 of 5 item selected .

Note : When I tested this on Window Inbuilt Screen reader like narrator it read properly and also JAWS also read same behavior as Narrator screen reader .

Please support if possible .What we need to do ?

[beqabeqa473]

Hello.

I can confirm that NVDA has several issues if we are talking about qt framework. Even narrator works much better in this area.

On 10/11/22, Aksh Singh @.***> wrote:

Hi Team I am facing one issue with NVDA Screen Reader when the focus goes to the Listview in Qt /QML Application it is not reading 1 of 5 ,Let's total 5 item in the list and current item is 1 . then it should read name of the item and then read selected and then 1 of 5 or 1 of 5 item selected .

Note : When I tested this on Window Inbuilt Screen reader like narrator it read properly and also JAWS also read same behavior as Narrator screen reader .

Please support if possible .What we need to do ?

-- Reply to this email directly or view it on GitHub: https://github.com/nvaccess/nvda/issues/8604#issuecomment-1274628357 You are receiving this because you are subscribed to this thread.

Message ID: @.***>

-- with best regards Beqa Gozalishvili Tell: +995593454005 Email: @.*** Web: https://gozaltech.org Skype: beqabeqa473 Telegram: https://t.me/gozaltech facebook: https://facebook.com/gozaltech twitter: https://twitter.com/beqabeqa473 Instagram: https://instagram.com/beqa.gozalishvili

[LeonarddeR]

Could you please provide some examples to test with?

[cppqtdev]

Hi leonardder , you can test with this qt qml application : https://drive.google.com/file/d/1FQFkaLHinkJ0KIOo_RrTk7fm6GRywUQs/view?usp=sharing

[shoogle]

Could we discuss new problems in new issues please?

I know it's tempting to post here because a Qt developer is subscribed, but the risk is that they will unsubscribe if all the comments are about NVDA bugs that happen to affect Qt software, rather than things that are actually the fault of Qt.

If you create a new issue and it does turn out to be on the Qt side rather than the NVDA side, you could leave a single comment in this thread linking to it to let subscribers know that it exists, but could we avoid having the discussion here please?

[ethindp]

@shoogle I'm pretty sure that these problems are a combination of NVDA and QT. Both need to do their part. QT needs far more attention from NVDA if this problem is to be resolved.

[shoogle]

@ethindp

1. It was said above that JAWS and Narrator are reading Qt lists properly, so it appears (at first glance at least) that this is purely an NVDA problem, so there is no reason to notify Qt developers at this stage.

2. Even if it does turn out to require some action on the Qt side, it is still a new problem that has very little to do with the original topic of this issue. It would be better to address it in its own dedicated issue so that people searching for "NVDA doesn't say list item numbers" (or similar) would be more likely to come across it.

[ethindp]

@shoogle Your first argument is correct, or would be, if it weren't for the fact that Orca, on Linux, has similar problems regarding QT accessibility in certain contexts. A standard QPlainTextEdit, for example, cannot be navigated without modifications of some kind (I am looking into this). On Windows, edit boxes of any kind, bar QLineEdits with single-line entries, cannot be navigated at all, at least with NVDA. If it weren't for the fact that both Orca and NVDA both suffer this problem, I would agree with you that this is purely an NVDA issue. However, it is clearly a problem with both NVDA and QT, or for all we know it could purely be a QT issue. Tracking down exactly what causes this problem will be difficult but is not impossible. Your second argument would be applicable if that was the only problem that NVDA and QT have, but it isn't. I think that the reason people continue commenting here is not just because a QT developer is subscribed to this issue but because its easier just to consolidate all of them into a single issue. Perhaps it might be better to create separate issues for each QT-specific issue, then create a central issue where we can have a list of all the issues collected and their status via a task list.

[shoogle]

@ethindp, what do QLineEdits have to do with the ListView problem raised earlier? You seem to be talking about something else entirely, which further illustrates the need to use separate issues to avoid confusing people.

Qt provides an abstraction layer for the underlying accessibility frameworks, which are very different on Windows and Linux. If Orca and NVDA exhibit similar problems then that could indicate a Qt problem, but not when all the other Windows-based screen readers are working properly with Qt, which appears to be true in the case of the ListView problem.

Perhaps it might be better to create separate issues for each QT-specific issue, then create a central issue where we can have a list of all the issues collected and their status via a task list.

Yes please! I don't mind if this is used as the central issue, but I don't think the back-and-forth discussion about whether a particular problem is or is not on the Qt side should take place here.