making flightGear accessible with NVDA screen-reader

[geoffmcl]

[Flightgear-devel] making flightGear accessible with NVDA screen-reader

Josh Kennedy <joshknnd1982@gmail.com> 28/02/18 03:35

Hello,

I am currently a eurofly pilot.

http://www.stefankiss.sk/modules.php?name=eurofly&file=downloadcenter&lng=en

But I would also like to be able to fully use and play, and fly all planes in the flightGear game. Could developers on this email list please make an interface module between flightGear and the open source NVDA screen reader which would make the game fully playable by visually impaired people who use screen readers?

www.nvaccess.org

I would really like to play a more realistic flight simulator. If NVDA could fully interface with flightGear and all aircraft, this would be wonderful!

Thanks

Josh

Sent from Mail for Windows 10

Torsten Dreyer <torsten@t3r.de> 28/02/18 09:45

Looks like this is Windows only - sorry, there is not much I can do (not using Windows at all). Probably somebody else on Win want to give it a try?

Torsten --

Torsten Dreyer

Edward d'Auvergne <true.bugman@gmail.com> 28/02/18 10:00

Josh, is there an equivalent free software screen reader that works on all operating systems? Even if it were lower quality, that might make accessing the accessibility application easier for all FlightGear developers to experiment with.

Cheers,

Edward

Daniel Wolak <danielwolak97@gmail.com> 28/02/18 12:34

Hi,

I'm not Josh but being blind as well I can answer this question. There unfortunately isn't a screenreader for all platforms. There are however speech modules that I believe work on all opperating systems. I'm thinking espeak primarily, and maybe festival, although I'm not sure if there's a windows binary.

Also, for windows in particular there's the microsoft speech api(sapi).

For output on windows tolk is the option I personally use. It is primarily windows based, but can output to most screenreaders including nvda, jaws, sapi and several more.

Cheers,

Daniel

Josh Kennedy <joshknnd1982@gmail.com> 28/02/18 17:17

Windows has NVDA. Mac has voiceover which is free, and Ubuntu Linux has Orca screen reader which is also free.

Josh Kennedy <joshknnd1982@gmail.com> 28/02/18 18:38

No. there is

NVDA for windows.
Voiceover for mac
Orca for Linux
Talkback for android

Each is specific to that OS.

Thorsten Renk <thorsten@science-and-fiction.org> 02/03/18 07:55

As a start, it would be great to have some tags in the interface, although as far as I can see at least the home screen is mostly accessible. It would also be nice to have an ability to read out the gages. (...) However, it would be nice if there could somehow be accessible output that announces the state of that as I interact with it.

That at least seems a simple matter of adding an option to the tooltip function to forward the tip string to the text to speech feature - so whenever you hover over a tooltip-equipped element long enough such that the tip pops up, you'd hear the content of the tip being read out.

Probably no more than a few lines added to the Nasal code (and some care in defining tooltips that don't come out garbled...)

• Thorsten

Edward d'Auvergne <true.bugman@gmail.com> 02/03/18 09:51

On 2 March 2018 at 00:18, Daniel Wolak danielwolak97@gmail.com wrote:

Evening,

Thanks for the response. I'm going to put a bit of background here if I might, to describe how we used to be able to access fsx.

For a great many years, there was a virrtual copilot called itsyourplane, herein refered to as simply iyp. This acted as an interface between the user and the aircraft, although it used speech recognition. It allowed control of most aircraft systems, and had specific checklists ETC. It also had the ability to read out certain information including things such as speed, heading altitude ETC.

As a start, it would be great to have some tags in the interface, although as far as I can see at least the home screen is mostly accessible. It would also be nice to have an ability to read out the gages. As far as a fully detailed description, it would be nice down the line somewhere, but I'd say that it's not 100% required for succesfull opperation. The key is to be able to keep an eye on the gages, as those are crucial in my book. Now personally, I've not been able to have a play about with the main interface itself, yet, although I do have some time and shall do some of that this evening. An interesting little thing I noticed as I was looking, once you select fly now in the main interface and the aircraft loads, screenreaders don't read out the menu bar. I'm only on windows, so am not sure what the state on mac/linux is. I'm not sure how they are layed out, but generally pressing alt brings up the menu bar. So let's say that I want to go to help>tutorials. Press alt, the menu pops up. I then arrow over to help, enter the menu and then select tutorials. I'm not sure if this is by design to prevent people from having the menus popping up or it's something on my end. Jumping back to how we interact with sims, I think the key is to have spoken feedback. An example is the magnetos. Let's say that I want to start up the default aircraft. As I'm looking at the wiki, it tells me to press } three times to set on both... Fantastic, excelently detailed. However, it would be nice if there could somehow be accessible output that announces the state of that as I interact with it. Just some general thoughts. Sorry I can't be more specific, but I don't really have the coding chops for something like this. Sure, I've done some python, but this is a whole different animal.

Do let me know if there are any specifics that would be useful, or if there's anything that you would like me to elaberate on further.

This is an interesting review about this It's Your Plane software in "blind pilot mode":

https://www.flightsim.com/vbfs/content.php?14197-Yes-Blind-People-Can-Become-Flight-Simmers

The video at the end is useful demo (but strangely text at the start of the video is not blind-friendly). Using your voice to communicate with FlightGear might be a little difficult, but there would be other uses for such new infrastructure (for example Thorsten's Alouette III copilot). Mozilla DeepSpeech as a 3rd party addition would be one open source option [1], but I don't know much about this area. This might need a lot of infrastructure work though.

Regards,

Edward

[1] https://github.com/mozilla/DeepSpeech

merspieler <merspieler@airmail.cc> 20180304

Hi Daniel,

Daniel Wolak:

although it used speech recognition. It allowed control of most aircraft systems, and had specific checklists ETC. It also had the ability to read out certain information including things such as speed, heading altitude ETC.

Right now I'm working on a speech recognition based copilot. Right now it includes voice commands for gear, flaps (inclusive speed checks) and checklists. Assistant for blind persons wasn't directly my intention when creating it but it might be a point to start with.

It's based on mycroft ( https://mycroft.ai/ ) which is currently not available for windows so you'd need to run flight gear on Linux or have a Linux machine running parallel to your Windows one.

If you're willing to assist me, I could add a mode for blind persons which suits their need (I have no idea what's needed to fly blind).

regards,

merspieler

[geoffmcl]

Josh Kennedy <joshknnd1982@gmail.com> 20180305 14:49

If you all would like me to do this I will… I will give you a first-hand audio demonstration of what is needed for blind people to fly planes.

I will make a recording of myself playing the eurofly flight simulator. And I will give detailed descriptions of what I am doing as I do it, and will set a voice that you all can understand. Just give me a day or two for my new headphones with 3d recording microphone to arrive. Once I get that in the mail I will promptly make a recording of myself playing eurofly with detailed descriptions of how blind people fly planes. This way perhaps something similar could be implemented into flightGear for all planes. I will provide a google drive link for the mp3. And for best results when listening to my audio demonstration, I recommend you use headphones or good stereo speakers.

Josh

Josh Kennedy <joshknnd1982@gmail.com> 20180305 14:53

We don’t need speech recognition to fly planes. I will put up a detailed demo of me flying a boeing 737 in eurofly in a day or so.

[ashleygrobler04]

Hi, I reelly want to get in to flying flight gear. Is there no way this awesome sounding flight sim could be made accessible for visualy impaired people using screen readers? Like reading the on screen instructions or what so ever?