search

libretro / RetroArch

Cross-platform, sophisticated frontend for the libretro API. Licensed GPLv3.
http://www.libretro.com
GNU General Public License v3.0
10.33k stars 1.84k forks source link

Accessibility: Have TTS speak GUI elements on focus #9661

Open devinprater opened 5 years ago

devinprater commented 5 years ago

First and foremost consider this:

Description

Now that Retroarch can speak in-game text using TTS, this feature should be extended to the GUI. Since we already know the text of the UI, like the main menu, game menu and such: GUI, this can simply be spoken using the TTS, no screenshot needed. I am willing to pay for the implementing of this feature as a bounty.

Of course, this should be a toggle, and could have a key command on Pc, Mac, and Linux, or a gamepad gesture, or exact set of gestures, to enable it, so that blind users can play games using Retroarch.

Expected behavior

RetroArch should speak the item in focus in the RetroArch UI, to help with accessibility by blind users.

Actual behavior

When retroarch runs, a blind user cannot navigate the main menu, to get cores, load cores, or otherwise control the program, except from the menu bar, which doesn't have all functions. Furthermore, the in-game menu isn't accessible either. This is why Retroarch should speak this itself.

Steps to reproduce the bug

  1. [Run a screen reader, like Narrator on Windows, VoiceOver on Mac, or Orca on Linux.]
  2. [Open Retroarch.]
  3. [Navigate Retroarch's screen with arrow keys, or the Tab key....]

Bisect Results

Version/Commit

You can find this information under Information/System Information

Retroarch 8.0

Environment information

BarryJRowe commented 5 years ago

I've looked into this. The best approach seems to be to use the hooks the screen reader uses to get what to read out, instead of hooking into the TTS that the AI Service uses. In that case there would be no need for a toggle button to turn it on or off. I should be able to work on this some time soon.

devinprater commented 5 years ago

Wouldn’t this need rewriting of the UI though? Or do you mean using the screen reader API’s, like the NVDA controller client? Or are there ways to tell OS’s that a custom control is a menu, and populate that menu? If so, that would be perfect.

On Nov 4, 2019, at 11:45 AM, Barry Rowe notifications@github.com wrote:

I've looked into this. The best approach seems to be to use the hooks the screen reader uses to get what to read out, instead of hooking into the TTS that the AI Service uses. In that case there would be no need for a toggle button to turn it on or off. I should be able to work on this some time soon.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/libretro/RetroArch/issues/9661?email_source=notifications&email_token=ADUMTTXHG3OVULZN5Q4IDL3QSBNS7A5CNFSM4JGNNVEKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDADNRY#issuecomment-549467847, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADUMTTU4NVWAM2VYOMPUQSDQSBNS7ANCNFSM4JGNNVEA.

BarryJRowe commented 5 years ago

@devinprater Accessibility APIs are what I mean. They're implemented in the OS, so that screen readers can read what's in the current application. Program authors write to these APIs so the screen reader doesn't have to figure out what's on the screen.

I haven't used these before, so I'll have to figure out what the best options are in terms of cross-platform compatibility.

Edit: The Qt library enables this automatically (usually) if you're using it to make a GUI. This page gives a good description of how it works: https://doc.qt.io/qt-5/accessible.html

devinprater commented 5 years ago

I’m not sure there is a way to use API’s cross platform. Microsoft has https://docs.microsoft.com/en-us/previous-versions/windows/desktop/dnacc/new-accessibility-model-for-microsoft-windows-and-cross-platform-development https://docs.microsoft.com/en-us/previous-versions/windows/desktop/dnacc/new-accessibility-model-for-microsoft-windows-and-cross-platform-development But it doesn’t look to be “cross platform” as the title says. MacOS has https://developer.apple.com/accessibility/macos/ https://developer.apple.com/accessibility/macos/

Linux has ATK's git repository https://gitlab.gnome.org/GNOME/atk

iOS has https://developer.apple.com/accessibility/ios/ https://developer.apple.com/accessibility/ios/

Android has https://developer.android.com/guide/topics/ui/accessibility https://developer.android.com/guide/topics/ui/accessibility

One way I can think of is to make a UI using a web app type system, like Electron, although hopefully not using that exact system. Or, maybe the UI Automation is cross platform after all, or can be translated into cross platform code from Windows or something.

On Nov 4, 2019, at 1:03 PM, Barry Rowe notifications@github.com wrote:

@devinprater https://github.com/devinprater Accessibility APIs are what I mean. They're implemented in the OS, so that screen readers can read what's in the current application. Program authors write to these APIs so the screen reader doesn't have to figure out what's on the screen.

I haven't used these before, so I'll have to figure out what the best options are in terms of cross-platform compatibility.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/libretro/RetroArch/issues/9661?email_source=notifications&email_token=ADUMTTQ2F4RNJNAUNYTIDD3QSBWZ7A5CNFSM4JGNNVEKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDALG6Q#issuecomment-549499770, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADUMTTUBM64O6UKVLIHAUSDQSBWZ7ANCNFSM4JGNNVEA.

BarryJRowe commented 5 years ago

Every OS has a different API, but I would like to find a solution that can use a single library that is cross-platform and abstracts away the native APIs. Qt for instance will use UI Automation on Windows, macOS Accessibility on macOS, and AT-SPI for linux. The accessibility stuff there is designed for applications made using Qt, which is not what we would do here, we would just bypass and specify exactly what to read out instead.

devinprater commented 5 years ago

Ah, this would work well then. Hopefully there is something like this for all platforms Libretro supports… well except for the Switch and Wii, unfortunately those don’t have a screen reader.

On Nov 4, 2019, at 2:35 PM, Barry Rowe notifications@github.com wrote:

Every OS has a different API, but I would like to find a solution that can use a single library that is cross-platform and abstracts away the native APIs. Qt for instance will use UI Automation on Windows, macOS Accessibility on macOS, and AT-SPI for linux. The accessibility stuff there is designed for applications made using Qt, which is not what we would do here, we would just bypass and specify exactly what to read out instead.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/libretro/RetroArch/issues/9661?email_source=notifications&email_token=ADUMTTWE5JPRLLZB64N2W4LQSCBQXA5CNFSM4JGNNVEKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDAT2OI#issuecomment-549535033, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADUMTTRYO5JCOU6DWGJ4EIDQSCBQXANCNFSM4JGNNVEA.

BarryJRowe commented 4 years ago

9768

Had to revert to self-voicing, but it should be customizable enough to hook into a screen reader with a little outside work. Waiting on some changes for language support in windows, and refactoring suggestions before merging.

devinprater commented 4 years ago

Thanks so much for this. If you need anything tested, I’d be glad to.

On Nov 21, 2019, at 4:12 PM, Barry Rowe notifications@github.com wrote:

9768 https://github.com/libretro/RetroArch/pull/9768

Had to revert to self-voicing, but it should be customizable enough to hook into a screen reader with a little outside work. Waiting on some changes for language support in windows, and refactoring suggestions before merging.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/libretro/RetroArch/issues/9661?email_source=notifications&email_token=ADUMTTV7SE6OVWA7CMVBBR3QU4BVZA5CNFSM4JGNNVEKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEE32U5I#issuecomment-557296245, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADUMTTWNYR5PDJDLYOEGANDQU4BVZANCNFSM4JGNNVEA.

BarryJRowe commented 4 years ago

Ok, I'll message you on twitter.