Make Dwarf Fortress accessible to blind players

[myk002]

This would be a major effort, and the first step is gathering requirements and potential leads. I'm creating this issue so we have a place to start assembling a plan and so we can have a basis for deciding whether this is even feasible.

There are good ideas in the forum thread: http://www.bay12forums.com/smf/index.php?topic=145179

[BlindGuyNW]

Some rough thoughts...

1. In an ideal world the game could communicate/speak through the screen reader directly, similar to this Stardew Valley accessibility mod. FAiling that, the terminal is a good way to prototype things and at least get readable output quickly.
2. Most of the game UI is keyboard driven (in the non-Steam version anyway), and we could leverage that. Is it possible to vocalize menu options as they are spoken? Perhaps for a start the terminal UI could get a list of menu options and output them.
3. A core problem is figuring out how to handle the map. I would prefer a UI similar to the way a sighted person would navigate--I don't want to oversimplify or dumb down things if it can be avoided.
4. A good start is vocalizing the selected tile under the cursor so that I can perhaps do designation and other tasks which require selecting specific places.
5. Embarkation and site selection and such are kind of special cases of menu accessibility.

Just some rough ideas to begin with. A lot of the stuff we need is already available in memory, I suspect, and the issue is how to present it. I don't have enough experience with the game to talk much about it in depth yet.

[BlindGuyNW]

Suggestion: There are a few speech synth libraries, such as Tolk. These will abstract away a lot of the screen reader communication issues, making it easier to send arbitrary text without worrying about lower-level concerns.

[BlindGuyNW]

Another repository for interesting accessibility ideas, a mod for Factorio

[BlindGuyNW]

Some points from a discussion tonight.

The map UI can be broken down into the ability to ittterate over lists of things; dwarves, buildings, enemy units etc. The key is to be able to move from one item to another quickly, and get important information either automatically or without much effort.

It is important to have access to coordinates or some other means to tell where map objects are relative to one another.

A map viewer of some kind would be useful to explore tile by tile, possibly with configurable range, so the position of the audio cursor moves in smaller or larger "steps." This could supplement the ability to list items.

[20k]

I'd be interested on working on this, I'd previously written off this being possible because I've always heard you need to use native OS widgets to get screen reader functionality, but it looks like with Tolk/etc it means there's solid cross platform support for the kind of API we'd need to do this

Currently I'm working on a complete UI re-implementation for various reasons, and given this it would be possible to entirely replace certain menus as-required with ones that are potentially much more friendly, and integrate screen reader support into the UI reimplementation in general. And then where menus are not re-implemented, fallback to a mode of reading the active menus in the most friendly way possible - which as far as I know should be possible.

I suspect it would be a good idea in my opinion for the sighted and screen-reader UIs to not diverge unless necessary (ie we shouldn't build a second non visual UI system for menus, we should annotate and redesign-if-necessary menus so that everyone's got the same ui layout)

Given that factorio and stardew valley both have support to some degree, it does seem well past time that we tried to put in some basic support into DF

One of the biggest bottlenecks I suspect is the lack of detailed knowledge on how to get this to work and the requirements. For my own reference, I've been watching some videos of the factorio accessibility mod with some extremely useful discussions in the discord as well - as far as I can tell one of the key requirements is the ability to iterate through nearby things at a point and get specific information. This video was helpful for figuring out what's going on

https://www.youtube.com/watch?v=5XHdjHDtebU&t=735s

While most menus should be straightforward (in 0.47), it seems like the key element will be iterating over things relative to the cursor in a way that lets you consume the sometimes large amount of map information that df chucks at you efficiently via a screenreader. It was also mentioned by zkline that distances and coordinates between things are important, to help get oriented

A lot of the immediate information you want in DF is dependent on intent, eg in a military situation you may want to read out enemy type, or in a labour situation you may want to start reading out dorf attributes. I suspect being able to place the screenreader into a mode where it priorities different kinds of information being read out in a user controllable way would be helpful to digesting large amounts of useful information, and hotkeys to navigate through the various attributes on a creature. Eg if you hit ctrl-i, it'll by default read out Name, type, hostile status, contents of their inventory from weapon to armour downwards, or if you hit ctrl-z (for status) it'll go Name, type, hostile status, and then read out injuries. But if you wanted to swap back and forth between the two, it could be i to get a focused creatures inventory, and z to read out health information, l to read out labours etc

I suspect default-most-relevant information is going to be very dependent on playtesting

On a less fun note, this post is all written with relatively vanilla 0.47 + potentially a few menu reworks in mind - 0.50 is fundamentally significantly more challenging in this respect. The UI in general is solely designed to be mouse driven, and many things have no keyboard shortcuts at all, eg you can't even navigate the main menu. A lot of work would have to go on on the dfhack end of things to make it work, and that's assuming that the reverse engineering process goes smoothly. Its likely going to be months before 0.50 is at a place where the UI will be able to be automated in any significant fashion. Given that it also lacks any concept of keyboard focus, we'd also need to build the ability to tab through tabs, navigate ui elements with the keyboard (!), and maintain accurate ui state in general on top of the DF UI

Eg dfhack needs to understand the concept of a tab bar as a whole unit to be able to navigate through it correctly. I also don't know that that unit of information is even conceptually used by Toady, they could simply be a fairly randomly ordered and rendered set of ad-hoc buttons. Its also not necessarily clear whether or not the process of submitting UI elements as a conceptual unit (eg render a checkbox with this title) will be accessible after reverse engineering, or if we'll simply get the sequence of steps to render a checkbox (ie set up state, specify a square, check if some state is true, specify an X etc). That's one for the RE people, but the UI submission system is as far as I know is effectively a total black box to people reverse engineering at the moment. If we can't hook the UI submission system, a straightforward approach in 0.50 is essentially impossible

This overall presents the biggest problem in my opinion, especially because it does not look like the 0.47 UI is coming back. In the 0.47 case I think its possible to make this work in an essentially vanilla game - with reimplemented UI only where necessary or as a helpful addition

In the 0.50 case I suspect the best approach might be to complete a UI reimplementation with built in screen reader support rather than trying to work with the existing UI is my first thought

[lizelive]

i would be interested in seeing the game working on an orbit slate 520. its 5 lines of 20 cells of brail (2x4 dot)

[BlindGuyNW]

I assume you are talking about five lines of 20 cells… Five lines of two cells is not going to display much of anything. I would love to see braille support as well, but it seems that that would require a lot more work than the already considerable project of native screenreader integration. Still, perhaps, the terminal UI could be repurposed…

[lizelive]

ah yes fixed. i think the ux would be less bad then voiice