Add Accessibility for Blind Developers Who Use Screenreaders

[ghost]

Operating system or device, Godot version, GPU Model and driver (if graphics related): Operating System: Arch Linux, might be true for blind users on other operating systems Godot Version: 2.1.4, may apply to all Issue description:

The Godot game editor does not work with the Orca screenreader. I was expecting to at least be able to use Orca's flat review with the editor application, but nothing happened; it is equivellent to turning off the monitor. Steps to reproduce: Install the GNOME desktop environment, along with the Orca screenreader. MATE will also work. With Orca running, open the Godot application. There will be no speech output from Orca. Link to minimal example project:

[ndarilek]

I'm sorry, I don't want to turn this into a general support thread, but in this case I don't know if I've done something wrong and the editor is flagging it with a warning or if I've genuinely made a mistake. I'm kind of doing a bit of a hybrid workflow where I edit some things in the editor and then, with it shut down, edit some things in the files by hand. So it's possible I've done something wrong, though I'm not getting syntax errors.

I can't get 3-D audio playing, and put together this example to make things as simple as possible, but it doesn't work. My intent is to create a single 3-D spatial node with an audio stream rumble and listener as children, then play the rumble. This works in 2-D, but not in 3-D. Am I doing something wrong here?

Apologies again, I asked in the forum without response, and text tutorials are hard to come by. Sometimes I don't know whether I'm hitting a Godot bug, or if the editor is flagging an error and my plugin isn't exposing it.

Thanks for any help. Other than this audio issue, I'm making some decent progress on Asteroids-style navigation with a spoken interface. There are still wrinkles, but working with the Godot editor as a totally blind developer is doable so far.

[ndarilek]

I've broken accessibility support out into a separate ScreenReader node that can, theoretically, be added to any SceneTree to make in-game UIs accessible. This makes my plugin not only an editor accessibility enhancement, but an equivalent to Unity's accessibility plugin that has been used in many shipping games. Of course, functionality is still lagging behind, but to the best of my knowledge, Unity's toolchain is not yet accessible so we're ahead in that regard.

[Faless]

I can't get 3-D audio playing, and put together this example to make things as simple as possible, but it doesn't work. My intent is to create a single 3-D spatial node with an audio stream rumble and listener as children, then play the rumble. This works in 2-D, but not in 3-D. Am I doing something wrong here?

You need to add a 3D Camera to the scene. You can then put the listener as a sub-child of the camera and move the camera, or move the listener directly. That should fix the issue...

which I think is a bug, since no error/warning whatsoever is displayed in the editor when no camera is present and sound is not played.

There are still wrinkles, but working with the Godot editor as a totally blind developer is doable so far.

This is heart-warming to hear, thank you for your work in making Godot more accessible.

[ndarilek]

Ah, I thought the listener was enough. Can I add the camera and prevent it from rendering while still keeping whatever property makes the listener work? I do want to be able to render something to 2-D, just not a full 3-D scene. Or can I tweak the 3-D camera to render 2-D somehow--give it an orthogonal perspective maybe?

Thanks, been pouring through the sources trying to track that down.

[Faless]

Ah, I thought the listener was enough

I think it should be, it might be a bug, I don't have much knowledge in that area.

Can I add the camera and prevent it from rendering while still keeping whatever property makes the listener work?

You can put all the 3d stuff in a Viewport and it will not render (remember to enable the listener property).

I do want to be able to render something to 2-D, just not a full 3-D scene. Or can I tweak the 3-D camera to render 2-D somehow--give it an orthogonal perspective maybe?

You can render both 3D in 2D (via ViewportContainer) and 2D in 3D via (Mesh, Material, ViewportTexture). There is some documentation here: https://docs.godotengine.org/en/3.1/tutorials/viewports/viewports.html and demo projects here: https://github.com/godotengine/godot-demo-projects under the viewport subfolder.

[ndarilek]

OK, I got my stream playing! I've read lots of the docs already, and am also a bit limited by my inability to see what's actually being rendered, so I hope you don't mind a few more questions inline. I'm asking these elsewhere but this is the only place I'm getting any answers, and I'm sorry to anyone who feels spammed, but I'm almost at the point where I'm considering hiring a Godot consultant to help with this open source project, which I'd like to avoid due to limited funds. BUt my questions:

On 9/25/19 10:38 AM, Fabio Alessandrelli wrote:

You can put all the 3d stuff in a |Viewport| and it will not render (remember to enable the listener property).

Wait, I thought that was the whole point of a viewport? So it sounds like what you're saying I can do is:

 * Create a 2-D game like I currently am, letting it render to the default viewport that gets created.

  Create a separate* viewport with a camera as a child and a listener as a child of the camera.

And in that case the camera won't render anything even if the viewport is a child of the scene tree? Is that because its size is set to 0 by default, or is there some other reason I can process code on an invisible viewport? That's entirely counter-intuitive to me, but if that works then it would greatly simplify my work. I also want to make sure that the camera continues to process even if the viewport isn't visible.

Do my AudioStreamPlayer3D nodes need to be children of the viewport as well? I was actually looking through the engine source, but it wasn't immediately obvious to me that the listener needed to be a camera child, so I'm not sure how to tell if AudioStreamPlayer3D nodes need to be children of the same viewport.

Thanks again.

[ndarilek]

OK, hitting another odd issue I need help with. If I add a Raycast2D node, then try to set its collision mask property via the editor, the menu of layers pops up when I press Enter on the button. But pressing Enter on a layer doesn't seem to close the menu and select a layer. Enter works fine on other PopupMenus, which is what this seems to be.

Any ideas why this may be happening? And if not, could someone please point me to where this particular menu is implemented in the engine so I can investigate? Poked around in editor/ but it isn't immediately obvious, and I'm also uncertain what specific criteria would prevent some PopupMenu nodes from responding to Enter. Maybe something is working on _gui_input and blocking it?

Thanks.

[Faless]

OK, hitting another odd issue I need help with. If I add a Raycast2D node, then try to set its collision mask property via the editor, the menu of layers pops up when I press Enter on the button. But pressing Enter on a layer doesn't seem to close the menu and select a layer.

Sorry about the late reply. Yes, the PopMenu is not closed automatically in that case. You need to press Esc to close it. The idea, is that when you set the collision layer (which is a bitmask field) you might want to set more than one bit at once, so the popup stays open. The relevant code is setting PopupMenu.hide_on_checkable_item_selection to false. See: https://docs.godotengine.org/en/3.1/classes/class_popupmenu.html#class-popupmenu-property-hide-on-checkable-item-selection

https://github.com/godotengine/godot/blob/master/editor/editor_properties.cpp#L799

https://github.com/godotengine/godot/blob/master/scene/gui/popup_menu.cpp#L1145

[ndarilek]

Sorry, should have mentioned that I'd cracked this one already. I'm trying not to spam this issue overly much if I can avoid it. :)

Making some good progress, though most of it is on my game since the accessibility layer is far enough along. There's still plenty more to do, though.

Next question, is there some way to intercept and filter touchscreen interaction before it reaches Controls or other nodes? I'd like to start working on some sort of explore-by-touch support as found in VoiceOver for iOS or TalkBack for Android, and also as implemented in Unity's accessibility plugin. Essentially, I'd like to intercept touches so that double-tapping a control is needed to trigger it, and regular screen touches only serve to reveal the interface. Quick swipes in certain directions also work as Tab/Shift-tab. I don't know if I need viewport overrides, some sort of custom layer that I can use as a filter, etc. I'd rather not change how every single control behaves, instead just filtering what interactions reach them in a way that can be toggled on and off while a game is running.

Thanks.

[Faless]

Next question, is there some way to intercept and filter touchscreen interaction before it reaches Controls or other nodes?

You can use the Control._gui_input(event) in the root GUI element or even Node._input(event) (possibly even in the root viewport). You can then use SceneTree.set_input_as_handled() or Viewport.set_input_as_handled() to block them after checking the input type for e.g. event is InputEventScreenTouch.

Check out: Input event flow: https://docs.godotengine.org/en/3.1/tutorials/inputs/inputevent.html

_input method in Node. https://docs.godotengine.org/en/3.1/classes/class_node.html#class-node-method-input

set_input_as_handled() method in SceneTree. https://docs.godotengine.org/en/3.1/classes/class_scenetree.html#class-scenetree-method-set-input-as-handled

_gui_input in Control https://docs.godotengine.org/en/3.1/classes/class_control.html#class-control-method-gui-input

[ndarilek]

Thanks, lots to look over here. Here's another:

https://docs.godotengine.org/en/3.1/classes/class_itemlist.html#class-itemlist-method-select

"Note: This method does not trigger the item selection signal." Why is that? I'm having to override some keyboard-handling on widgets because, for instance, pressing down-arrow on the lowest item of a tree/list will skitter focus onto a neighboring widget instead of simply not advancing the selection. Same with ItemList. I'm also having to implement my own selection logic. One odd issue I'm hitting is in the file selector, where selecting an item in the list isn't updating the filename in the text field. I end up having to select the directory that the file is in, then type the filename into this field.

Given that I'm having to implement my own selection focus logic, I'm wondering if the fact that select doesn't fire the signal might be what's causing this? And that begs the question, why doesn't select fire the signal indicating that an item was selected?

Thanks.

[ndarilek]

OK, I'm to the point where I'm exporting a game that uses the accessibility plugin for its UI. This is posing a new challenge which I don't know how to address.

I have code that attempts to guess a label for some fields. Part of this algorithm involves traversing up through a node's parents, finding any EditorProperty instances, and returning their labels if any. This works if I run my game in the editor, or via a binary that has the editor built in. It fails with an exported binary, because EditorProperty isn't defined in binaries without the editor. Things I've tried:

if node.get_class() == "EditorProperty" doesn't work because I need the is check, not just class equality, and I'd rather not check whether the class name equals any descendant of that class's name.

if Engine.is_editor_hint() doesn't work because the failing code still runs in the exported binary.

Can I retrieve a class by its name into a variable, then perform logic based on whether the variable is null or not? I need to run an is check or its equivalent. Failing that, can I move this check into a separate context that isn't run in exported binaries, or evaluates to null? I still want this code to run generally, but would be happy to move the editor-specific checks out of the code path for exports.

For reference, here's the code I'm trying to make work:

func guess_label():
     var tokens = PoolStringArray([])
     var to_check = node
     while to_check:
         if Engine.is_editor_hint():
             print(to_check)
             if to_check is EditorProperty and to_check.label:
                 tokens.append(to_check.label)
             if (to_check is EditorProperty or to_check.get_class() == 
"EditorInspectorCategory") and to_check.get_tooltip_text():
                 tokens.append(to_check.get_tooltip_text())
             var label = tokens.join(": ")
             if label:
                 return label
         to_check = to_check.get_parent()

Thanks.

[ellenhp]

@ndarilek,

Would substituting an is_class check for the get_class check solve this problem? The docs for that method are here.

[ndarilek]

Yup, worked nicely. Thanks! The alternative I was considering would have been much messier.

[ellenhp]

Glad to hear it. I went back and read some of these posts and it's not entirely clear what the outstanding issues are and what's been resolved, but you did mention there is some difficulty with figuring out what's being rendered. I need this plugin for a project I'm working on so I'm open to helping out wherever you need a sighted developer for testing or resolving weird quirks. Feel free to contact me with what you need done or populate the issue tracker on your gitlab repos with bugs and feature requests you need help with and I'll tackle whatever I think I can manage. My email is ellen.h.poe@gmail.com

On Mon, Oct 14, 2019, 9:50 AM Nolan Darilek notifications@github.com wrote:

Yup, worked nicely. Thanks! The alternative I was considering would have been much messier.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/godotengine/godot/issues/14011?email_source=notifications&email_token=AAOY262NSXUDGOMDFEF5NPTQOSPOXA5CNFSM4EG4X6EKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBFRGQQ#issuecomment-541791042, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOY264T3T7KWWSTBIIJUFLQOSPOXANCNFSM4EG4X6EA .

[ndarilek]

We may be able to close this one soon, OTOH I've had lots of questions answered here that IRC/the forums haven't helped with. That's true again. :) Hopefully I've kept the S/N ratio high, but when I need help with something, it's generally above and beyond what I can find in a tutorial or book, and there are plenty of skilled folks watching this issue and helping me build this support out.

DId lots of work on this over the holidays. My godot-tts plugin now supports Android and HTML 5, and I've exported accessible games to both platforms. I also did some initial work on touchscreen accessibility. On my Linux desktop, UIs can now be explored accessibly on my $70 HDMI touchscreen. Simple touches speak the UI control being touched, a double-tap anywhere on-screen activates the last focused control, and a quick swipe right/left acts like Tab/Shift-tab and moves focus between elements.

Unfortunately, swiping doesn't work at all on Android, and I'd appreciate help figuring out why. More specifically, the swipes themselves are detected just fine. They then inject ui_focus_next or ui_focus_prev actions using code like this:

func press_and_release(action):
    var event = InputEventAction.new()
    event.action = action
    event.pressed = true
    get_tree().input_event(event)
    event.pressed = false
    get_tree().input_event(event)

func swipe_right():
    press_and_release("ui_focus_next")

func swipe_left():
    press_and_release("ui_focus_prev")

And those actions don't trigger on Android, even though the swipe_right/swipe_left functions run just fine. Any idea why that might be?

I plugged a keyboard into my phone, and Tab at least triggers ui_focus_next. So the action works in terms of being recognized, but as generated via the above code it doesn't seem to trigger. I also tried Input.action_press and Input.action_release, but that made things stop working even on the desktop. So clearly my code does something Input.action_press doesn't, but it doesn't do enough to make Android happy. I tried to find where the events were converted to actions, but it isn't clear to me whether or not these paths are platform-specific. Clearly they have to be since Linux/X11 and Android diverge.

Running out of ideas here and am open to suggestions. Thanks for the help so far.

[akien-mga]

Superseded by https://github.com/godotengine/godot-proposals/issues/983.

Note that this might be worth splitting in a separate proposal again. We're closing old proposals on this repository to encourage migrating them to the new godot-proposals tracker where we want all proposals to reside and be discussed.