Add Accessibility for Blind Developers Who Use Screenreaders

[ghost]

Operating system or device, Godot version, GPU Model and driver (if graphics related): Operating System: Arch Linux, might be true for blind users on other operating systems Godot Version: 2.1.4, may apply to all Issue description:

The Godot game editor does not work with the Orca screenreader. I was expecting to at least be able to use Orca's flat review with the editor application, but nothing happened; it is equivellent to turning off the monitor. Steps to reproduce: Install the GNOME desktop environment, along with the Orca screenreader. MATE will also work. With Orca running, open the Godot application. There will be no speech output from Orca. Link to minimal example project:

[akien-mga]

This would definitely be a welcome improvement, but I can already say that it will likely take a while to get implemented, if at all. Godot's editor uses its own GUI toolkit, which is the same used in games, and is fully independent from the native toolkits or cross-platform ones such as Qt or GTK+, which would have good screen reader support.

So to get this to work in Godot, it would have to be implemented from scratch in the core Controls. I don't know how hard this is, but probably non trivial.

[vnen]

I have actually thought about this but I don't even know where to start. Godot editor is very visual, you do pretty much everything with the mouse. We would need a person with visual impairment to give us consulting about this and guide us in this task, advising us with what is actually needed and how it should behave. Without an actual user to provide feedback, it's mostly a game of guessing to implement that.

[ethindp]

I would be happy to guide you guys in adding this. I'm blind myself. :) At least, I could refer you guys to frameworks (MSAA and UIA come to mind as possibilities, though UIA is preferable to me at least, since NVDA (https://github.com/nvaccess/nvda) (an open source screen reader) uses it) and beta test it as you guys go. In the meantime, is there an alternative godot editor or method of creating the game by hand, without the editor?

[MarianoGnu]

If this feature is implemented in the core of the engine itself then it would be possible to developeprs to make apps for blind people, not only allow blind developpers to work. +1 to the proposal

[UrbanFuturistic]

I'm glad to see this is being looked at and the only thing I have to add about this is that screen reader software is also use by people who have learning disabilities/difficulties such as dyslexia. As much as I hate the way these things sometimes leave people out, I think it could be argued that this broadens the case for such a development.

NV Access (NVDA) would be a good first place to contact as both lead developers are blind and so have a proper insight into what's necessary. Maybe a collaborative effort?

[francipvb]

Hi,

I think that this will be a huge effort. That can be done but considering native accessibility protocols an implementation made from scratch can be very problematic for users of all platforms.

If you plan to go forward with this, I recommend to you to use the QT model, an abstract accessibility event layer on top of the basic UI controls first and a platform dependent implementation.

Another approach is to use Text To Speech directly. This can be more platform-independent and so much easy to work on. There are some problems with this, too, but they are more related to the engine than the runtime environment.

Hope this can help you deside.

Cheers,

[ethindp]

Such a model would work too, if contacting the low-level accessibility APIs failed in some way. I 100 percent agree that NVAccess would be your best bet for getting the highest level of accessibility in. :)

On 5/18/18, Francisco R. Del Roio notifications@github.com wrote:

Hi,

I think that this will be a huge effort. That can be done but considering native accessibility protocols an implementation made from scratch can be very problematic for users of all platforms.

If you plan to go forward with this, I recommend to you to use the QT model, an abstract accessibility event layer on top of the basic UI controls first and a platform dependent implementation.

Another approach is to use Text To Speech directly. This can be more platform-independent and so much easy to work on. There are some problems with this, too, but they are more related to the engine than the runtime environment.

Hope this can help you deside.

Cheers,

-- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/godotengine/godot/issues/14011#issuecomment-390176511

-- Signed, Ethin D. Probst

[ndarilek]

As a partial solution, are there any docs, or even places to look, to implement a game by hand without using the editor? I've seen the command line docs, but I'm wondering if it's even practical/remotely enjoyable to work with the scene graphs and such by hand, thus completely eliminating the need for the editor.

As a more practical example, right now I'm implementing an audio-based Asteroids-like game. Asteroids emit sounds to mark their positions, and the player centers them in the audio field by turning/flying around. I don't particularly care about the visuals, but they'd be useful for debugging purposes so I can ask sighted folks if, say, my mental concept of my ship's orientation matches with the crude cylinders/spheres I'm using for debugging/collision detection. I'm wondering if it's even remotely possible to position a 5-unit-diameter sphere at 0,0, randomly position and move a bunch of 25-unit-diameter spheres, implement wrapping behavior, etc. by hand-coding it all without the need of the Godot editor. Or will it be a mess of editing non-human-friendly XML?

At the moment I am doing this more or less by hand in Rust with an ECS. That's not terrible, but I do wonder if I gain anything by leveraging an actual game engine. Ideally the Asteroids thing is just a proof-of-concept, and I'd eventually like to make richer audio games without having to build all this stuff up by hand. Whether Godot is that solution, I don't know...

Either way, I do think that implementing a full accessibiity interface to the UI toolkit is probably a bit much to ask. If the data files are mostly text but human-unfriendly, a middle-ground solution might be a DSL that exposes a simplified interface for the kinds of tasks I described above (positioning shape primitives, attaching sounds, adding logic to entities. etc.)

[vnen]

As a partial solution, are there any docs, or even places to look, to implement a game by hand without using the editor?

I think the easiest way (without hacking around) would be to generate the scenes procedurally via script. You could either do that at runtime or make a script that saves the scenes and resources to disk so you can add to the project. The initial scaffolding would require a project.godot file for the project settings (which can be an empty file made by the OS at first) and a main scene that runs at start (which can also be set and saved via script). Might be complex to manage, but it's possible.

Technically, you can make a script that runs directly from the command line and that script makes the whole Godot project for you (at least I can't think of anything that would prevent that). You can then run and export the game from the command line as well. You would need to open the project in editor at least once for importing the assets (and every time the assets change) but that could be solved by running the headless version of Godot, as some people are doing for use with Continuous Integration servers.

If the data files are mostly text but human-unfriendly, a middle-ground solution might be a DSL that exposes a simplified interface for the kinds of tasks I described above (positioning shape primitives, attaching sounds, adding logic to entities. etc.)

The text resource format is not much hard to read, but still it's probably simpler to just use GDScript to create stuff for the project, since you can use the base engine for writing and reading the format (avoiding compatibility issues).

Unfortunately, GDScript is not the best language for visually impaired people, since the white-space is important. I heard complains from a blind developer about Python because of this.

[ethindp]

Frankly, I think that the complaints about whitespace placement is most likely (excuse my language) bitchy wining. Whitespace is perfectly fine (I'm blind and am able to work with python very well, as well as other whitespace-oriented languages like F# and such). Those who complain about whitespace are usually those who are either unenthusiastic at the thought of programming (which raises the question of exactly why they're checking it out in the first place) or those who haven't even tried and who are just complaining to complain. I don't mean to be so inflammatory but it really pisses me off when I see people complaining about that. It just makes absolutely no sense. I've never heard of the headless version of Godot. Perhaps that could be used to create projects and interact with the engine as well? That might be a complete solution to our problems at the moment. Where can I find this version? (Or is it in the Godot package?)

On 5/24/18, George Marques notifications@github.com wrote:

As a partial solution, are there any docs, or even places to look, to implement a game by hand without using the editor?

I think the easiest way (without hacking around) would be to generate the scenes procedurally via script. You could either do that at runtime or make a script that saves the scenes and resources to disk so you can add to the project. The initial scaffolding would require a project.godot file for the project settings (which can be an empty file made by the OS at first) and a main scene that runs at start (which can also be set and saved via script). Might be complex to manage, but it's possible.

Technically, you can make a script that runs directly from the command line and that script makes the whole Godot project for you (at least I can't think of anything that would prevent that). You can then run and export the game from the command line as well. You would need to open the project in editor at least once for importing the assets (and every time the assets change) but that could be solved by running the headless version of Godot, as some people are doing for use with Continuous Integration servers.

If the data files are mostly text but human-unfriendly, a middle-ground solution might be a DSL that exposes a simplified interface for the kinds of tasks I described above (positioning shape primitives, attaching sounds, adding logic to entities. etc.)

The text resource format is not much hard to read, but still it's probably simpler to just use GDScript to create stuff for the project, since you can use the base engine for writing and reading the format (avoiding compatibility issues).

Unfortunately, GDScript is not the best language for visually impaired people, since the white-space is important. I heard complains from a blind developer about Python because of this.

-- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/godotengine/godot/issues/14011#issuecomment-391900905

-- Signed, Ethin D. Probst

[vnen]

@ethindp I won't argue about what is or isn't, since I'm not blind myself, but I'll ask you to tone down. People have different backgrounds and might have proper reasons to find it hard to work, while you learned to work with it. The person I know works as a developer for a living, I find hard to believe he "didn't try hard enough" (it's still second-hand for me though, so I won't argue about that).

I've never heard of the headless version of Godot. Perhaps that could be used to create projects and interact with the engine as well? That might be a complete solution to our problems at the moment. Where can I find this version? (Or is it in the Godot package?)

It is called the server platform and works on Linux only. It's not distributed (maybe it'll be once 3.1 is released), so you need to compile yourself to access it. This is just the regular Godot engine, but it has dummy drivers for video in order to not require a graphics card.

I'm not sure how much you can interact with it, likely you can't do it manually. Still, it can be used to run scripts and importing assets, so it's possible to make a project with it if you do everything with scripts (at least in theory).

[ndarilek]

I'll echo what @ethindp says in that indentation shouldn't be an issue. Most modern screen readers have settings to speak indentation automatically (I.e. "8 spaces def hello_world():`, and I know lots of prolyphic blind Python developers.

I wonder if there's even another middle-ground solution--a line-oriented command line tool that can manipulate the scene graph in real-time, drive another instance, and let you save its output. I'm thinking of an interaction style like:

$ godot-shell mygame.godot
Welcome to the Godot shell.

> new entity
Entity created with ID 0.
> add component 0 position
Component "position" added with ID 0.0.
> set component 0.0 [0, 0]
Component 0.0 set.
> save
Scene graph saved.
>

I'm then imagining a separate instance running in another window, displaying whatever the engine is currently outputting, playing sounds, etc. Other commands might spawn a GUI text editor to edit scripts.

Would this kind of workflow make sense if the aim is to create a more simplistic game? I.e. I don't really care what my player/world models look like, beyond them being simple shapes so I can mentally model them, attach sounds, then have a collision detection system that makes some sort of sense. I also think a CLI is a bit easier to prototype and experiment with than is a full-blown DSL.

I may start hacking on this if folks who know more than I do about Godot think it's viable. Been looking for a good CLI interface problem to sink my teeth into, and have a few Rust libraries I'd like to put through their paces. Is the file/data format documented anywhere?

[ethindp]

@George Marques, I apologize for my rude behavior... just letting that off my chest. Because it really pisses me off and seems like wining more than an actual truth, you know? @Nolan Darilek, your idea may have merit; would it be possible to code an alternative UI interface, perhaps using something like WXWidgets? If we used WX, it would solve all our accessibility issues in one go. (Personally, I've never fully managed to figure out WX.... looked way too complex for me.... heh :)).

On 5/25/18, Nolan Darilek notifications@github.com wrote:

I'll echo what @ethindp says in that indentation shouldn't be an issue. Most modern screen readers have settings to speak indentation automatically (I.e. "8 spaces def hello_world():`, and I know lots of prolyphic blind Python developers.

I wonder if there's even another middle-ground solution--a line-oriented command line tool that can manipulate the scene graph in real-time, drive another instance, and let you save its output. I'm thinking of an interaction style like:

$ godot-shell mygame.godot
Welcome to the Godot shell.

> new entity
Entity created with ID 0.
> add component 0 position
Component "position" added with ID 0.0.
> set component 0.0 [0, 0]
Component 0.0 set.
> save
Scene graph saved.
>

I'm then imagining a separate instance running in another window, displaying whatever the engine is currently outputting, playing sounds, etc. Other commands might spawn a GUI text editor to edit scripts.

Would this kind of workflow make sense if the aim is to create a more simplistic game? I.e. I don't really care what my player/world models look like, beyond them being simple shapes so I can mentally model them, attach sounds, then have a collision detection system that makes some sort of sense. I also think a CLI is a bit easier to prototype and experiment with than is a full-blown DSL.

I may start hacking on this if folks who know more than I do about Godot think it's viable. Been looking for a good CLI interface problem to sink my teeth into, and have a few Rust libraries I'd like to put through their paces. Is the file/data format documented anywhere?

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/godotengine/godot/issues/14011#issuecomment-392111005

-- Signed, Ethin D. Probst

[ndarilek]

Likewise re: WX. If I was going to do a GUI, I'd probably go with GTK since I'm on Linux, but I've never sat down to learn GTK or WX, and that would seem like one too many variables to play with here.

I think I'd initially start with a CLI that can start/stop a rendering Godot process and drive this headless version. If I did this in Rust, I'd probably build a struct and serde implementation to a subset of the format that the interface would support. With that as a starting point, it should be easy to build a GUI later or in parallel.

[ethindp]

That would work, yes; but how exactly are you going to do this in rust? I think rust would be perfect for this, but from what I've seen, the engine editor at least is written in C++. Actually, it seems the core is, too. So, since (according to https://doc.rust-lang.org/beta/nomicon/ffi.html) Rust can't interface with C++, we'd need to create a C interface. That in itself would be an absolute nightmare. :)

On 5/25/18, Nolan Darilek notifications@github.com wrote:

Likewise re: WX. If I was going to do a GUI, I'd probably go with GTK since I'm on Linux, but I've never sat down to learn GTK or WX, and that would seem like one too many variables to play with here.

I think I'd initially start with a CLI that can start/stop a rendering Godot process and drive this headless version. If I did this in Rust, I'd probably build a struct and serde implementation to a subset of the format that the interface would support. With that as a starting point, it should be easy to build a GUI later or in parallel.

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/godotengine/godot/issues/14011#issuecomment-392121156

-- Signed, Ethin D. Probst

[ndarilek]

I was thinking of editing the files directly, which seems more doable if you expose a more limited interface that grows over time (I.e. initially only expose basic shapes, rigid bodies, sounds, etc. then add more over time.) I'm looking at the format and it appears to be mostly INI, I.e.:

[node name="WallContainer" type="Node" parent="." index="0"]

editor/display_folded = true

Although I didn't know section names could be quite that free-form. :) I guess there's nothing in the INI spec, such that it is, to prevent that.

So as an MVP I'd want:

  Rigid dynamic/static 2-D bodies for collision detection and motion   A few basic shape primitives to mentally model the bodies, and represent them on-screen for visual feedback. Cylinders for characters, spheres, and some sort of blocky polygon for walls/grounds should be enough for now.   Spatial audio   Non-spatial audio for music/UI sounds  * Access to scripts, and the ability to add scripted behaviors to any of the above that support it

I don't want to call that a small feat, but if I could figure out the nodes needed to implement that featureset, then create a text user interface for building that structure and then calling into the Godot tooling for export/rendering, we might be onto something. There are lots of node types, but if those are documented in any sort of machine-readable way, it might be possible to eventually tap some sort of Rust code generator or write a macro to replicate the same UI en mass...but it's best to figure out what that UI looks like before reaching for a macro to autogenerate it. :)

If I'm right about everything being INI-based, I might put together an MVP PoC to see where this goes. To be clear, I'm not aiming for a general-purpose editor use case here, just some way to build audio games without necessarily coding the whole thing by hand, and leveraging an actual game engine to do the cross-platform export.

But as fun as this brainstorming is, I gotta get to work and finish out the week. I'll put up a repo sometime soon and see where it goes. I generally prefer GitLab because of its integrated CI, but I could be convinced to stay on GitHub if folks are opposed.

[vnen]

The file format is described (at least partially) in the documentation: http://docs.godotengine.org/en/latest/development/file_formats/tscn.html

However, making a completely new tool for this is a lot of work and is bound to become obsolete quite easily. At least you should use GDNative as a bridge (which is a C interface for the C++ engine), that would give access to all information about classes and their properties. As much as I appreciate the effort, I think you first need to understand a bit what Godot has to offer, so you don't reinvent all the wheels.

If you really want to do with Rust, you can try the bindings for GDNative (though I can't attest it's stability): https://github.com/GodotNativeTools/godot-rust

Personally, I would try to do something with the Godot core itself (maybe as a C++ module), since you can then access all the API easily, including all the resource savers and loaders, so you don't need to worry about file formats.

---

If you're going for a CLI interface, it should be based on arranging the scene tree, which is the most important part of any Godot game. Designing this process beforehand is probably more important: creating an easy way to understand how the tree is laid out and tools to move nodes to a specific tree position should be the cornerstone of the design. If you get that, you don't even need to limit the feature-set, since pretty much anything can be done with the tree.

After that, you need a way to set properties of the individual nodes, including the resources that are used to give then a proper functionality (e.g. a CollisionShape2D node needs a Shape2D resource to give its shape). Scripts are also resources, so they can be set via this interface as well (though you might want to give them a special status). This would substitute what the Inspector is in the GUI.

Also, I would make commands as close to GDScript as possible. For instance, you want to create a Sprite, you could simply do Sprite.new(), which is the same you would do in a script. This way the learning curve for someone already used to scripting would be less steep.

For audio, Godot uses players and buses. There are players for positional audio (both 2D and 3D) and a regular one for non-positional audio, which are all nodes in the tree as well. Surround is also supported (though I never used it). To work with the buses, you need to interact with the AudioServer, to set volume, mute or solo them, and add effects. You might be able to use the AudioBusLayout resource directly, but I'm not sure how much of it is exposed to the external API. The effects are just resources, so they can be edited by the same interface.

---

All of this in my humble opinion, of course. I have used Godot for quite a long time now, so I might be somewhat biased.

[ndarilek]

Hey, thanks, this is helpful. I'm not particularly wedded to messing around with the data files directly. Happy to learn how this might work--most of the getting started docs I encounter teach by building games, and no one assumes you'd need to build your own tools first. :)

I haven't touched C++ in nearly 2 decades, so I'll likely stick with Rust and deal with a less complete API or file issues if I hit any instability. Does gdnative support reading/writing project, scene and other files directly? Or is it meant to support writing games in C and calling into the engine? I did a brief grep through the Rust repo for "project" and didn't find anything. The examples seem focused on building games directly. If you'd not mind giving me an entrypoint for where I'd start to build a project and populate it programatically, I'll read up on that and move further work to my own repo.

Thanks again for humoring me. :)

[vnen]

GDNative is meant to make games, essentially replacing scripts. But as I said before, you can do pretty much anything with scripts, and GDNative uses the same API.

[francipvb]

Hi,

For scene or alike and complex objects, you can use a format like this file.

In this case, it is just for a map for a project called audioquake, but it can be an example. P.D: Don't try to build this, it may not work.

For GUI, you can also use QT5.11, I saw that it includes many improvements for accessibility. In this case, you will only need to add some sort of accessibility support for hard custom controls like the camera.

Cheers,

[ethindp]

The format for audio quake maps is very limited. I personally don't like it.

On 5/25/18, Francisco R. Del Roio notifications@github.com wrote:

Hi,

For scene or alike and complex objects, you can use a format like this file.

In this case, it is just for a map for a project called audioquake, but it can be an example. P.D: Don't try to build this, it may not work.

For GUI, you can also use QT5.11, I saw that it includes many improvements for accessibility. In this case, you will only need to add some sort of accessibility support for hard custom controls like the camera.

Cheers,

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/godotengine/godot/issues/14011#issuecomment-392214077

-- Signed, Ethin D. Probst

[ndarilek]

Wait, I'm being an idiot. You said at the beginning of this thread that you didn't need the editor at all, except for when importing assets. I think it would have been more correct for me to ask if I needed the GUI to create games. I'm happy running them from the GUI, I just want to develop them on the console. And it looks like I can.

So I guess what I want is a "Godot from scratch" series that assumes you can't/don't want to use the editor, similar to how you'd go about writing a game with pygame or another high-level library. I imagine once I've figured out how to create a .project file that just launches a script, and maybe a script that just opens a window and spins a cube or something, I can apply that upward. I doubt such a thing exists, so maybe I'll try it myself and blog the process. Then maybe I'll build a CLI prototyping tool later, once I know what I might want to prototype without code.

[ethindp]

You know... I wonder if we can somehow make the engine not only an engine with an editor and such, but a library too? Like, your given the C++ libraries (static or dynamic libs) and can build your game however you like, using Godot for all the game engine stuff (i.e. audio and such). If this was possible we could forego the editor entirely. So you could add assets as, say, .caf files, and then, at runtime, you'd compile them in memory as compiled asset data files (we could just call them CADFs). So your game would look like, at startup:

//... // Compile assets godot::assets::LoadAll({asset, asset, asset}); // load each asset individually or Godot::Assets::LoadAllAssetsFrom("assets"); // load all the assets at once from a folder. // Compile all the assets auto assets = Godot::Assets::GetAllLoadedAssets(); bool success = Godot::Assets::Cadf::CompileAllAssets(assets); if (success) { // assets compiled } else { // error }

Granted, this is my style of coding, and I doubt its the way the engine is written. But if it were possible to do that, it would be epic. Or we could just forego asset compilation at all, just load all of 'em when we need 'em, or all of 'em at startup.

On 5/25/18, Nolan Darilek notifications@github.com wrote:

Wait, I'm being an idiot. You said at the beginning of this thread that you didn't need the editor at all, except for when importing assets. I think it would have been more correct for me to ask if I needed the GUI to create games. I'm happy running them from the GUI, I just want to develop them on the console. And it looks like I can.

So I guess what I want is a "Godot from scratch" series that assumes you can't/don't want to use the editor, similar to how you'd go about writing a game with pygame or another high-level library. I imagine once I've figured out how to create a .project file that just launches a script, and maybe a script that just opens a window and spins a cube or something, I can apply that upward. I doubt such a thing exists, so maybe I'll try it myself and blog the process. Then maybe I'll build a CLI prototyping tool later, once I know what I might want to prototype without code.

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/godotengine/godot/issues/14011#issuecomment-392224709

-- Signed, Ethin D. Probst

[vnen]

I think it would have been more correct for me to ask if I needed the GUI to create games. I'm happy running them from the GUI, I just want to develop them on the console. And it looks like I can.

You don't need to GUI to run games. In fact, the editor just creates another Godot process with the arguments to run the game (with all the debugging options as well). Running the game is a non issue: you can simply run Godot from the project's directory without arguments, the default is to run the project.

The editor is pretty much a Godot "game" itself. It uses the available nodes and extends them. The part about importing, it's not that you need the GUI, you just need the importing tools that comes with the editor. Essentially you just put all assets in the project folder (in any structure your like) and when you open the editor it'll import everything.

"Importing" in this case means storing metadata (if you want to loop and trim audio, or apply filter to an image, etc.) and sometimes converting to a format that Godot understands and can use (like importing a model to a Godot mesh). The import process can only be done by the tools build (i.e. the editor), even if the headless version.

When you export the game, only the imported assets are packed, the original ones are ignored. For the game itself this process is transparent: you can treat the assets as if they were in the original location, the loaders will know to get the imported version.

Just like the editor is just a game, you can make your own "game" that is actually a tool to make games. I mean, you can even create a script that extends MainLoop or SceneTree and run directly via terminal with:

$ godot -s my_script.gd

The problem is that you want a true CLI tool, Godot won't make it easy. Even when running a script it'll open a window, and everything sent via stdin will be ignored.

---

I wonder if we can somehow make the engine not only an engine with an editor and such, but a library too? Like, your given the C++ libraries (static or dynamic libs) and can build your game however you like, using Godot for all the game engine stuff (i.e. audio and such).

While the editor and the engine are made for each other, you're not entirely dependent of the editor to make the game. You cannot split Godot into pieces though, you need to take the whole package (although you can disable most of the modules without problem). Since the editor is a Godot game, it's technically possible to replace the code by your own game. That would mean creating the scenes on the fly with code.

Godot is not meant to be used as a library or framework, and likely won't ever be, but you can hack around it and make a MainLoop implementation that does not even use nodes at all, instead just call the servers directly. Of course, you would need to build the engine from source, but I imagine this is the least of the concerns.

Any of this is technically possible, and maybe not that hard, but it's venturing into uncharted territory.

[ndarilek]

Sorry for being noisy, folks.

So I've since done a fairly deep dive into Godot and have read lots of the documentation. Normally I don't do this for a non-library tool like this because it's kind of a long path towards the desired result--essentially building all the tools in your woodworking shop before you can cut a single sheet of plywood. :)

I'm slowly coming around to the view that the editor itself might be made accessible by essentially building a mini screen reader in GDScript. This has the added benefit of making the game UIs themselves accessible. It isn't without precedent, either. See this for a Unity UI accessibility plugin.

In looking at the Control class, there do appear to be signals exposed for focus enter/leave. There are also signals for input events, which I hope includes arrowing around text entry fields and the like. It'd be limited for a generic OS-level screen reader, but just might work for something domain-specific.

I've started on something here. Unfortunately, I can't add this plugin to my project because I need the inaccessible editor to add the accessibility plugin. :) If anyone could add this plugin to the project and submit a PR with the project.godot changes, that'd be helpful. Essentially what I'm going to do is hook the node_added (or whatever it's called) signal on SceneTree, intercept any nodes that descend from Control, and connect to their focus_entered signal. Once there, I'll start adding logic to print out presentation messages for each node type that will eventually be piped to an as-of-yet-unwritten TTS API. Please let me know if there are any obvious reasons why this wouldn't work.

And how would I make this a non-editor plugin that would work with any Godot UI, including those in non-editor games?

Hopefully we can eventually move discussion to this other repo and stop spamming this issue. :)

[ethindp]

If I understand you right, if this were to work, we could choose to include it in our games or not, right? Is there any way we could add it to the editor but not to the projects we create, or to remove it from the project but add it into the editor?

On 5/26/18, Nolan Darilek notifications@github.com wrote:

Sorry for being noisy, folks.

So I've since done a fairly deep dive into Godot and have read lots of the documentation. Normally I don't do this for a non-library tool like this because it's kind of a long path towards the desired result--essentially building all the tools in your woodworking shop before you can cut a single sheet of plywood. :)

I'm slowly coming around to the view that the editor itself might be made accessible by essentially building a mini screen reader in GDScript. This has the added benefit of making the game UIs themselves accessible. It isn't without precedent, either. See this for a Unity UI accessibility plugin.

In looking at the Control class, there do appear to be signals exposed for focus enter/leave. There are also signals for input events, which I hope includes arrowing around text entry fields and the like. It'd be limited for a generic OS-level screen reader, but just might work for something domain-specific.

I've started on something here. Unfortunately, I can't add this plugin to my project because I need the inaccessible editor to add the accessibility plugin. :) If anyone could add this plugin to the project and submit a PR with the project.godot changes, that'd be helpful. Essentially what I'm going to do is hook the node_added (or whatever it's called) signal on SceneTree, intercept any nodes that descend from Control, and connect to their focus_entered signal. Once there, I'll start adding logic to print out presentation messages for each node type that will eventually be piped to an as-of-yet-unwritten TTS API. Please let me know if there are any obvious reasons why this wouldn't work.

And how would I make this a non-editor plugin that would work with any Godot UI, including those in non-editor games?

Hopefully we can eventually move discussion to this other repo and stop spamming this issue. :)

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/godotengine/godot/issues/14011#issuecomment-392255752

-- Signed, Ethin D. Probst

[ndarilek]

I don't know.

I did figure out the format for adding a plugin to the project.godot file. I now have a function that should be adding focus_entered/mouse_entered handlers to each Control and prints a message when they have focus. Unfortunately, I'm not actually getting any of these events, nor am I sure how to get the actual node that originated them.

Nothing in this is editor-specific, though, and it should automatically detect the addition of any UI control anywhere in the scene tree and, eventually, implement a screen reader for it.

[ndarilek]

OK, here's where I am after some intermittent holiday weekend hacking:

1. Repository moved here. I'd rather work on it under my potential game development company identity, but I'll be releasing the plugin under the MIT license if it goes anywhere, so collaboration welcome.
2. In theory, I'm generating classes with accessibility-specific code whenever a Control is added to the tree.
3. Was told on IRC that tab/shift-tab behavior is already defined in the UI, but when the editor launches, nothing has focus so tab/shift-tab don't do anything until something gets focus.
4. I'm trying to set an initial focus, and I seem to be doing so with some random LineEdit widget, but tab/shift-tab doesn't seem to move the focus. This could be because they're being captured by the LineEdit control.
5. Setting an initial focus to anything else doesn't seem to work--at least, not according to my focus_entered callback.

I'm having a tough time with this because I'm new to the engine, but I'm coming at it from a direction which most newbies don't take. :) If anyone can help me crack this initial focus puzzle, I think I can make lots of rapid progress. Once I can use tab/shift-tab to navigate between controls, I can look at which properties each exposes and start creating an accessible presentation for them. I'm also happy to file issues against the engine itself--I just don't know what specific behaviors to ask for just yet.

Thanks.

[vnen]

You probably need to set the focus_mode of the Control. I believe the default is to not accept focus, except for the ones that need it (like text and button controls). You may also need to set the focus neighbors, I'm not sure how well the TAB works without it (or if it works at all).

[ndarilek]

Thanks. I just started hacking around with FOCUS_MODE_ALL. With that, I can call grab_focus on previously unfocusable items.

Regarding neighbors, I read at http://docs.godotengine.org/en/3.0/classes/class_control.html:

    If the user presses Tab, Godot will give focus to the closest node to the right first, then to the bottom. If the user presses Shift+Tab, Godot will look to the left of the node, then above it.

That's exactly what I'd implement if I did so myself, so it sounds like the default is perfect.

If you have a moment, could you please fire up the editor in a new project, maybe one with an empty project.godot just to keep things similar, and see under what conditions tab/shift-tab work in the editor? I think the person who helped test things on IRC may have created their own layout, whereas what I want to do is test things directly in the editor. If tab/shift-tab don't work, then I have a specific bug to file. The current belief under which I'm operating is that they don't work initially, but do after an item is clicked on.

And, interestingly enough, when I track focus_exited I do get an event when I press tab. Unfortunately, I never get another focus_entered, so once my focus leaves a control, it never returns to one.

Thanks for your help.

http://docs.godotengine.org/en/3.0/classes/class_nodepath.html#class-nodepath

[vnen]

If you have a moment, could you please fire up the editor in a new project, maybe one with an empty project.godot just to keep things similar, and see under what conditions tab/shift-tab work in the editor?

Tab does nothing at first opening. If I click somewhere to give focus, then it starts cycling around the controls, but it does not look like all of them are focused. The tabs of the dock containers does not seem to be focused, which means some parts of the interface can't be accessed via keyboard only.

[ndarilek]

Thanks. This issue is spinning out of control a bit. I'm going to file another regarding brainstorming ideas around editor focus.

[ndarilek]

Not sure what to do, I filed #19230 but there hasn't been much activity. I also posted on the forum and was advised to file an issue. Thoughts on where to ask for help next? I can't see whether or not my addon is actually setting focus, whether or not Tab/Shift-tab moves it if so, why I get a focus_exited event on every keypress even if it isn't a navigation key. I don't want to be annoying, but I feel like I could make a lot of progress contributing with 10 minutes of help from an experienced Godot developer. Thought I'd poke this issue and bump #19230, which has a more focused (no pun intended) list of questions.

Thanks.

[ndarilek]

Just wanted folks to know that this repository is making rapid progress. I can now:

• tab/shift-tab around the editor, getting feedback about which control has focus. Currently has semi-intelligent presentation of Button and LineEdit. More to come soon.
• Arrow left/right in LineEdit controls, getting feedback about which character has focus. Additional LineEdit support to come once I see how rapidly PRs are handled. I just submitted one adding a caret_moved signal.

Help welcome, though be advised that this addon now requires a version of Godot with unmerged PRs. Specifically, see https://gitlab.com/lightsoutgames/godot-accessibility/issues/1 for a checklist of all submitted PRs and their merge status.

If you'd like to help but don't want to step on any toes, a great way to do so would be building a TTS module. Right now this addon just prints to the console. What I'd like is an API like this:

TTS.speak("hello, world.", interrupt = True)
TTS.stop() # Interrupts speech if in progress
TTS.rate = 200 # We'd need to coordinate some sort of rate algorithm between engines.

Anything more complicated, like voices, can wait for later. If someone wants to help and creates such an API, I can probably make it work under Linux and Android. Otherwise I'll get to this eventually, though the screen reader takes priority for me.

[ethindp]

If you can tell me how you implement new GDScript APIs, I can implement that for windows at least.

On 6/12/18, Nolan Darilek notifications@github.com wrote:

Just wanted folks to know that this repository is making rapid progress. I can now:

• tab/shift-tab around the editor, getting feedback about which control has focus. Currently has semi-intelligent presentation of Button and LineEdit. More to come soon.
• Arrow left/right in LineEdit controls, getting feedback about which character has focus. Additional LineEdit support to come once I see how rapidly PRs are handled. I just submitted one adding a caret_moved signal.

Help welcome, though be advised that this addon now requires a version of Godot with unmerged PRs. Specifically, see https://gitlab.com/lightsoutgames/godot-accessibility/issues/1 for a checklist of all submitted PRs and their merge status.

If you'd like to help but don't want to step on any toes, a great way to do so would be building a TTS module. Right now this addon just prints to the console. What I'd like is an API like this:

TTS.speak("hello, world.", interrupt = True)
TTS.stop() # Interrupts speech if in progress
TTS.rate = 200 # We'd need to coordinate some sort of rate algorithm between
engines.

Anything more complicated, like voices, can wait for later. If someone wants to help and creates such an API, I can probably make it work under Linux and Android. Otherwise I'll get to this eventually, though the screen reader takes priority for me.

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/godotengine/godot/issues/14011#issuecomment-396575401

-- Signed, Ethin D. Probst

[ndarilek]

I don't know. I'm blind and learning all of this too, so if I'm learning how to build the TTS module then I'm not building the screen reader module. :) I'll get to learning that eventually, but my hands are full pushing GDScript as far as I can, then tweaking the engine when I can't get something out of GDScript. This whole process is very much like when I began building my Android screen reader. I didn't know what was possible, just started hacking on things, and only really got a sense for what I was doing a few months in.

And my assumption is that you'll need GDNative, unless GDScript offers some sort of FFI. I guess you'd build a C/C++ module that calls Windows' speech APIs and exports the functions to GDScript. Then I'd take that module and make it work on Linux/Android using Speech-dispatcher and Android's native TTS. I suppose I could also do web as well using Javascript's TTS API. If GDScript does offer FFI to C/C++, please do let me know.

Thanks!

[ethindp]

I'll look around and see if I can export some functions using the Tolk library to communicate with screen readers.

On 6/12/18, Nolan Darilek notifications@github.com wrote:

I don't know. I'm blind and learning all of this too, so if I'm learning how to build the TTS module then I'm not building the screen reader module. :) I'll get to learning that eventually, but my hands are full pushing GDScript as far as I can, then tweaking the engine when I can't get something out of GDScript. This whole process is very much like when I began building my Android screen reader. I didn't know what was possible, just started hacking on things, and only really got a sense for what I was doing a few months in.

And my assumption is that you'll need GDNative, unless GDScript offers some sort of FFI. I guess you'd build a C/C++ module that calls Windows' speech APIs and exports the functions to GDScript. Then I'd take that module and make it work on Linux/Android using Speech-dispatcher and Android's native TTS. I suppose I could also do web as well using Javascript's TTS API. If GDScript does offer FFI to C/C++, please do let me know.

Thanks!

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/godotengine/godot/issues/14011#issuecomment-396669298

-- Signed, Ethin D. Probst

[ndarilek]

That looks almost perfect. Let me know if you have a GDNative module working for this. If not, I'll implement something for Linux in a month or so, perhaps.

[vnen]

If you don't want to make it as a module (which requires recompiling the whole engine), the only way to access external libraries is to use GDNative. You can't do FFI with GDScript.

[ethindp]

I'll just add it to the GDScript global functions. That fine?

On 6/12/18, George Marques notifications@github.com wrote:

If you don't want to make it as a module (which requires recompiling the whole engine), the only way to access external libraries is to use GDNative. You can't do FFI with GDScript.

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/godotengine/godot/issues/14011#issuecomment-396736512

-- Signed, Ethin D. Probst

[ndarilek]

I think it'd make better sense as a static class with members. I.e.:

TTS.speak("Hello, world", interrupt = True)
TTS.stop()

That, at least, is how other modules seem to do it.

A quick google search reveals a nice GDNative tutorial complete with a sample project using scons. If you start from there, I should be able to take that and add speech-dispatcher support. Not sure how one adds, say, JS to interface with the WebSpeech API, but one thing at a time.

[ethindp]

Well crap. I already committed and uploaded it to my fork of the repo as a part of GDScript. If you visit https://github.com/ethindp/godot, you can see how I did it. I know, I went outside the norm, but Tolk, even in C/C++, is not a class-based library. So right now (I think) you can invoke it like so:

tts_load()
tts_output("Hi", false);
tts_unload()

In order for this to work with a screen reader, you'll need some extra files (along with the screen reader):

• JAWS, Window Eyes, ZoomText, System Access: Nothing, works through COM.
• NVDA: NVDAControllerClient32.dll and NVDAControllerClient64.dll (https://www.dropbox.com/s/7txp6iyi65sx12z/nvdaControllerClient32.dll?dl=1 and https://www.dropbox.com/s/y0pdyxhos31hv9n/nvdaControllerClient64.dll?dl=1)
• Dolphin screen readers: Dolapi32.dll (https://www.dropbox.com/s/m04mpzi7z6bfu5i/dolapi32.dll?dl=1)
• Microsoft Speech API (MSSAPI/SAPI): sapi32.dll and sapi64.dll (https://www.dropbox.com/s/7czg0rt9ht9yloq/SAAPI32.dll?dl=1 and https://www.dropbox.com/s/e2fxek89p6muz2h/SAAPI64.dll?dl=1) Alternatively, you can download all of them here: https://www.dropbox.com/sh/aatj7myhczyxs5u/AAA6K5aAZWis9uAF4CsNz_-Za?dl=1. I haven't tested this, I just incorporated it. Sorry if that was a bad idea or not. :) I'm not sure how we'll get tts_speak/tts_output/tts_braille to work since they require a wchar_t. This exports the following functions: tts_load(): Initializes Tolk by loading and initializing the screen reader drivers and setting the current screen reader driver, provided at least one of the supported screen readers is active. Also initializes COM if it has not already been initialized on the calling thread. Calling this function more than once will only initialize COM. You should call this function before using the functions below. Use tts_is_loaded to determine if Tolk has been initialized. tts_is_loaded(): Tests if Tolk has been initialized. Returns true if Tolk has been initialized, false otherwise. tts_unload(): Finalizes Tolk by finalizing and unloading the screen reader drivers and clearing the current screen reader driver, provided one was set. Also uninitializes COM on the calling thread. Calling this function more than once will only uninitialize COM. You should not use the functions below if this function has been called. tts_try_sapi(bool): Sets if Microsoft Speech API (SAPI) should be used in the screen reader auto-detection process. The default is not to include SAPI. The SAPI driver will use the system default synthesizer, voice and soundcard. This function triggers the screen reader detection process if needed. For best performance, you should call this function before calling tts_load(). Parameters: trySAPI: whether or not to include SAPI in auto-detection. tts_prefer_sapi(bool): If auto-detection for SAPI has been turned on through tts_try_sapi(), sets if SAPI should be placed first (true) or last (false) in the screen reader detection list. Putting it last is the default and is good for using SAPI as a fallback option. Putting it first is good for ensuring SAPI is used even when a screen reader is running, but keep in mind screen readers will still be tried if SAPI is unavailable. This function triggers the screen reader detection process if needed. For best performance, you should call this function before calling tts_load. Parameters: preferSAPI: whether or not to prefer SAPI over screen reader drivers in auto-detection. tts_detect_screen_reader(): Returns the common name for the currently active screen reader driver, if one is set. If none is set, tries to detect the currently active screen reader before looking up the name. If no screen reader is active, NULL is returned. Note that the drivers hard-code the common name, it is not requested from the screen reader itself. You should call tts_load once before using this function. tts_has_speech(): Tests if the current screen reader driver supports speech output, if one is set. If none is set, tries to detect the currently active screen reader before testing for speech support. You should call tts_load once before using this function. tts_has_braille(): Tests if the current screen reader driver supports braille output, if one is set. If none is set, tries to detect the currently active screen reader before testing for braille support. You should call tts_load once before using this function. tts_output(string, bool): Outputs text through the current screen reader driver, if one is set. If none is set or if it encountered an error, tries to detect the currently active screen reader before outputting the text. This is the preferred function to use for sending text to a screen reader, because it uses all of the supported output methods (speech and/or braille depending on the current screen reader driver). You should call tts_load once before using this function. This function is asynchronous. Parameters: str: text to output. interrupt: whether or not to first cancel any previous speech. tts_speak(string): Speaks text through the current screen reader driver, if one is set and supports speech output. If none is set or if it encountered an error, tries to detect the currently active screen reader before speaking the text. Use this function only if you specifically need to speak text through the current screen reader without also brailling it. Not all screen reader drivers may support this functionality. Therefore, use tts_output whenever possible. You should call tts_load once before using this function. This function is asynchronous. Parameters: str: text to speak. interrupt: whether or not to first cancel any previous speech. tts_braille(string): Brailles text through the current screen reader driver, if one is set and supports braille output. If none is set or if it encountered an error, tries to detect the currently active screen reader before brailling the given text. Use this function only if you specifically need to braille text through the current screen reader without also speaking it. Not all screen reader drivers may support this functionality. Therefore, use tts_output whenever possible. You should call tts_load once before using this function. Parameters: str: text to braille. tts_is_speaking(): Tests if the screen reader associated with the current screen reader driver is speaking, if one is set and supports querying for status information. If none is set, tries to detect the currently active screen reader before testing if it is speaking. You should call tts_load once before using this function. Returns true if text is being spoken by the screen reader, false otherwise. tts_silence(): Silences the screen reader associated with the current screen reader driver, if one is set and supports speech output. If none is set or if it encountered an error, tries to detect the currently active screen reader before silencing it. You should call tts_load once before using this function. Returns true on success, false otherwise. Enjoy! :)

On 6/12/18, Nolan Darilek notifications@github.com wrote:

I think it'd make better sense as a static class with members. I.e.:

TTS.speak("Hello, world", interrupt = True)
TTS.stop()

That, at least, is how other modules seem to do it.

A quick google search reveals a nice GDNative tutorial complete with a sample project using scons. If you start from there, I should be able to take that and add speech-dispatcher support. Not sure how one adds, say, JS to interface with the WebSpeech API, but one thing at a time.

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/godotengine/godot/issues/14011#issuecomment-396753704

-- Signed, Ethin D. Probst

[ndarilek]

Tolk itself doesn't have to be class-based to export a class-based interface. Take a look at the tutorial for a pure C-based example of creating a GDScript class. First hit on Google for "gdnative tutorial".

I won't be able to use this because I already have an engine fork with a custom signal. I'd like to avoid tweaking the engine unless necessary, and only in the minimal ways necessary to build accessibility in addons.

Good start, though.

[ndarilek]

Ah, to hell with it, I'm just going to take a stab at it. :) Except it's going to be in Rust, so that's that. Looks like there's a Tolk Rust binding. I can't do the Windows port, so you'll have to fill in those blanks yourself.

[ethindp]

There's a Tolk rust binding? Where? I couldn't find it... I'll look for it. :)

On 6/12/18, Nolan Darilek notifications@github.com wrote:

Ah, to hell with it, I'm just going to take a stab at it. :) Except it's going to be in Rust, so that's that. Looks like there's a Tolk Rust binding. I can't do the Windows port, so you'll have to fill in those blanks yourself.

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/godotengine/godot/issues/14011#issuecomment-396768693

-- Signed, Ethin D. Probst

[ndarilek]

It's the first hit when googling "tolk rust" for me. Additionally it shows up on cargo search tolk..

In any case, another Godot question. If I have a GDNative module that exposes a TTS class to GDScript, how do I ensure that class is globally available? I've created my .gdnlib file, but the tutorial at http://docs.godotengine.org/en/3.0/tutorials/plugins/gdnative/gdnative-c-example.html just shows a graphic when creating the .gdns file. What does a .gdns file look like so I can create one by hand? Looks like I load the class in via preload rather than just make it globally available, is that accurate?

Thanks.

[vnen]

if I have a GDNative module that exposes a TTS class to GDScript, how do I ensure that class is globally available?

Maybe it's possible to add it as an autoload singleton. In fact, there's a place for GDNative singletons in the editor, so maybe there's an even better way to create them. However there's no documentation about it and I have no idea how it works.

just shows a graphic when creating the .gdns file. What does a .gdns file look like so I can create one by hand?

After following the instructions it generates a file like this:

[gd_resource type="NativeScript" load_steps=2 format=2]

[ext_resource path="res://bin/gdexample.gdnlib" type="GDNativeLibrary" id=1]

[resource]

resource_name = "gdexample"
class_name = "gdexample"
library = ExtResource( 1 )

[ndarilek]

An update:

I'm thinking of either abandoning the project entirely or forking the engine. As a bit of context, I spent 4-5 years building Spiel, a fairly major Android screen reader, during a time when Android's accessibility APIs weren't very great. I like to think that I'm somewhat knowledgeable about what makes a good accessibility API, and am also sensitive to the performance needs of the underlying OS and user experience. Ultimately I abandoned the project in large part because it was difficult to build atop an incomplete accessibility API, and I had to resort to hack upon hack to make Spiel useful.

Godot is kind of the same way. There are actually enough UI signals for a reasonably sophisticated accessibility API. But I'll need a few more for an accessibility addon to have enough information. Specifically, #19522 would have made things easier, but I was asked to refactor that into addon-specific code. Now #19814 is also being debated. I understand not wanting to add a million signals, but I'm trying to be sensitive to that. All of my changes should stay confined to the UI layer, and I can't imagine this having much of an effect unless someone is playing a typing game where every last bit of performance is needed to render an EditLine at 120 FPS. :) Even so, the core is going to need some additions if folks really want this to happen.

And at the end of the day I just wanted to make games, so if I have to rigorously defend every PR on my own without much support from the folks who'd like to see this happen, or if I'm building all the addons by myself, I'm not really doing that. Not trying to stir stuff up, I just wanted to leave an update here either to inspire some sort of process discussion on how we might make this happen if it is really wanted, or to let anyone considering taking it up know that there may be some pushback.

Sounds like #19840 may be discussed at an upcoming PR meeting. If folks genuinely do want Godot to be more accessible, I'd appreciate more advocates than me at that meeting. I don't even know when said meeting is, or if it won't conflict with actual work meetings I need to attend for $dayjobs. :) I'm just realizing I can't do this alone, and am going to need some help from any interested parties.

Thanks.

[malcolmhoward]

@ndarilek If you're still interested in working on this, I'd like to help.

[ndarilek]

I know it's been a while, but I'm going to try picking this up again. Folks have recommended one large patchset with all needed accessibility fixes to the engine, so I'll give that a shot and hope no one asks me to split it up. :)

But I'm hitting some limits in therms of what I as a blind person can do. At the moment I have a speech interface that lets me tab around the editor, interact with menus, interact with/change some project settings, etc. I'm getting to the point where I'm ready to try taking on some timple text-based tutorials, but haven't quite unlocked enough of the interface. Could someone please answer the following:

• Where is the input map tab under Project -> Project Settings? I see a tree of options--config, run, editor, editor plugins, etc. but no input map. I'm trying to get to a list of action names so I can see what is available. I'm wondering if this is in a part of the interface that I haven't made accessible yet, though I'm surprised it isn't in the category tree.
• Can I programatically trigger an action? In particular, some tutorials tell me to right-click a node, but I don't see a way to do this from the keyboard. I'd like to bind the Menu key to right-click so I can follow some of the steps in this tutorial.
• Alternatively, is there a keyboard-friendly way to right-click, and a list of keyboard shortcuts anywhere? Maybe my issue is that Menu works, but I don't have enough engine accessibility exposed to know that.

And @malcolmhoward, if your offer to collaborate is still open a year later, I'm down. I'm willing to give this one more shot. I see you tried merging some Festival support. If you're interested in continuing that work, I have a Rust-based TTS plugin that currently supports Speech Dispatcher under Linux and Tolk under Windows. Would love to diversify that support (and to test the Windows Tolk support for that matter, since Linux is my primary platform.)

Thanks.