Requesting accessibility improvements, GUIs built with HISE are currently difficult to use with screen reader software

[ScottChesworth]

Hi,

I'm a blind producer, using Reaper with an accessibility extension called OSARA which bridges Cockos' API and screen reader software. Picture using the computer with keystrokes for all of your input, and having a synthetic voice telling you about what's happening on screen as you move around the GUI and tweak controls, that's the basics of the UX.

I'm posting here because although JUCE has an accessibility API, it seems that GUIs built with HISE aren't outputting accessible controls in line with that yet. Last year @GoranRista posted on the forum to report some bugs with keyboard navigation among other accessibility issues, there doesn't seem to have been any obvious movement on these yet and now he's getting close to releasing a major update for his instruments, which already have interest in place from blind/visually impaired drummers and producers. I'm hoping I can get dialog started here and - ultimately - some implementation that makes HISE GUIs more usable by more people.

Here's Goran's post on the HISE forum: https://forum.hise.audio/topic/7426/plugins-for-the-visually-impaired-users-att-all-developers?_=1709335582815

Reposting key findings here too:

• BUG: Slider values can not be adjusted using arrow keys, nor page up/down, home/end.
• BUG: Buttons are read as "checkbox" element type. Can you please change this to say "button"?
• Can we have the ability to exclude specific elements from being read by screen readers, such as icons? For example, I have many icons behind buttons that blind users must scroll through to access functional elements.
• A nice touch would be a visual highlight of some sort for the element which has been selected in the tab order. It would improve overall usability.

@GoranRista has already invested a bunch of hours going through the GUIs of his entire series of instruments improving the labelling of controls. However, without some work on the points mentioned above, the effort he's put in won't result in much of an accessibility improvement yet, because blind/visually impaired users are still going to need advanced trickery like navigating the object hierarchy and mouse emulation just to interact with his GUIs. Requisite nerdy knowledge of that kind is a huge barrier to entry, particularly when you consider that it's in stark contrast to the UX of sighted folks with many of the instruments built on HISE.

Would love to get your thoughts on this @Christoph-Hart. If accessibility is a new topic for you, feel free to ask absolutely anything. Would also be happy to meet up for a chat about it, can demo some of the tech for you, show you how some good and bad JUCE GUIs compare to what Goran is getting from HISE right now.

Thanks in advance for any and all effort you can muster! :)

[ScottChesworth]

Tagging my man @timtam here too, a good source for accessibility advice if needed, also in Germany like Christoph.

[christoph-hart]

Hi Scott,

thanks for your input. If there are any low-hanging fruit things I can easily modify, I'm happy to change things, but I'm afraid I don't have the time at the moment to fully embrace the Accessibility API of JUCE at the moment.

BUG: Buttons are read as "checkbox" element type. Can you please change this to say "button"?

A quick search in the JUCE codebase yielded a place where I could hack around to change this, but can you explain to me why this is necessary (from a naive look that sounds like cosmetics to me).

BUG: Slider values can not be adjusted using arrow keys, nor page up/down, home/end.

You can assign a key press to any UI element in HISE so unless it requires some default handling this should work. The problem with adding this as "default" behaviour is that most DAWs consume the arrow key strokes as change track shortcut and I wouldn't want to mess with that.

Can we have the ability to exclude specific elements from being read by screen readers, such as icons? For example, I have many icons behind buttons that blind users must scroll through to access functional elements.

Sure I can add a property to control this for each element but I would think that it can reuse the property of whether an element has attached a keyboard shortcut to it (because if it hasn't, there's no point focussing that).

A nice touch would be a visual highlight of some sort for the element which has been selected in the tab order. It would improve overall usability.

Again, I can't make this the default as this would highly interfer with existing interface designs (so I would rather say that this should be a tool that is possible to implement from each developer. Maybe a function like Content.getFocusedComponent() would be nice so that everybody can add some hinting in their graphics code.

[ScottChesworth]

A quick search in the JUCE codebase yielded a place where I could hack around to change this, but can you explain to me why this is necessary (from a naive look that sounds like cosmetics to me).

It's necessary because screen reader software relies on the correct roles to be assigned to tell users about it. Think about the behavioural patterns of the button types you encounter every day, each checkbox has two states available, a simple pushbutton control doesn't, in a group of radiobuttons only one can usually be ticked. Right now when screen reader users navigate HISE GUIs, all of those different controls are reported as being checkboxes, so users have no clue what behaviour patterns to expect when they interact with each control.

[ScottChesworth]

The problem with adding this as "default" behaviour is that most DAWs consume the arrow key strokes as change track shortcut and I wouldn't want to mess with that.

Sure, I understand why it's important to tread carefully here. But even when I deliberately send all keyboard input to the GUI of Goran's instruments, sliders can't be adjusted with the keyboard yet. If you did want to be able to control a slider with keystrokes, where would you assign that?

[ScottChesworth]

Can we have the ability to exclude specific elements from being read by screen readers, such as icons? For example, I have many icons behind buttons that blind users must scroll through to access functional elements.

Sure I can add a property to control this for each element but I would think that it can reuse the property of whether an element has attached a keyboard shortcut to it (because if it hasn't, there's no point focussing that).

Hold on, that's a dangerous assumption. What about things like text prompts, tooltips etc. Those don't need to be focusable with a keystroke per se, but there are many GUIs that would be way more confusing if suddenly they became invisible.

[ScottChesworth]

Maybe a function like Content.getFocusedComponent() would be nice so that everybody can add some hinting in their graphics code.

Sounds good, thanks!

[davidhealey]

Goran's instruments, sliders can't be adjusted with the keyboard yet. If you did want to be able to control a slider with keystrokes, where would you assign that?

I believe this can be achieved with a broadcaster. Just so I understand properly. The knob/slider has to have keyboard focus, and then the left/right cursor keys change the value, is that correct? Or should it be the up/down keys?

[ScottChesworth]

Convention would be for Tab and Shift+Tab to navigate through GUI controls until a slider gets focus, then Up and Down arrows to adjust it. Any advice appreciated.

[davidhealey]

I just had a look in the docs and it seems there isn't a keyboard broadcaster! I thought there was. I saw though that Christoph has just added a generic keyboard callback, so perhaps that can be utilised here.

[GoranRista]

Sorry for being a little late for the convo.

If I am not mistaken, the lowest-hanging fruits would be to name the buttons as buttons and be able to control the sliders using up/down arrows.

Since this affects not only blind users of my plugins, but also all plugins built on HISE and any blind person who may wish to use them, it would be very beneficial to implement this asap. I ask you, @christoph-hart to please, please, please find the time to implement this soon 🙏. I care about this so much that re-organizing all my plugins and renaming all my components (hundreds upon hundreds of them) was the very first thing I did for v2 Handy Drums. It took me several weeks to complete last summer.

I feel very strongly that if a person, who is unable to see, will go through all the challenges to use our software, the very least we can do is do whatever we can to make that process as easy as possible.

I can't wait to proudly proclaim that Handy Drums are now fully accessible to all! 😊

Thank you, Chris!