search

hrydgard / ppsspp

A PSP emulator for Android, Windows, Mac and Linux, written in C++. Want to contribute? Join us on Discord at https://discord.gg/5NJB6dD or just send pull requests / issues. For discussion use the forums at forums.ppsspp.org.
https://www.ppsspp.org
Other
11.04k stars 2.15k forks source link

PPSSPP UI elements not shown to accessibility services #11696

Open devinprater opened 5 years ago

devinprater commented 5 years ago

What happens?

Using the VoiceOver accessibility feature, available on all iPhones from 3GS and up, I cannot navigate the user interface of PPSSPP. As a blind person, I play video games, mainly fighting games like Soul Calibur, by using the sounds in game. PPSSPP works fine on Windows and Mac, where I can simply edit the INI file to change things, but I cannot find a way to do this on iOS. Since my device isn't jailbroken, I need to set the CPU method, but cannot do this without using the controls in PPSSPP.

What should happen?

The UI of PPSSPP should be easily navigated by VoiceOver, using either UIKit or other UI systems. All buttons onscreen should be shown to the accessibility service and clearly labeled with text.

What hardware, operating system, and PPSSPP version? On desktop, GPU matters for graphical issues.

iPhone X R running iOS12.1.3 (Public beta) running PPSSPP 1.6.3

unknownbrackets commented 5 years ago

I think it'd be awesome to support this.

It seems like there's no way to integrate with VoiceOver on iOS without either treating things as media closed captions (which doesn't seem like it would work), or entirely using iOS's layout system (UIKit, as you mentioned.) There seems to be support for custom controls, but they still have to be part of an iOS dedicated layout.

The unfortunate reality is that PPSSPP is not really "officially" supported on iOS - there aren't many active contributors to the iOS code. Jit doesn't work on latest iOS versions (as you mentioned), it crashes if exited in certain ways, and it's not tested nearly as much. PPSSPP is open source, and not enough iOS users are contributors.

That means even if someone did completely rebuild PPSSPP's iOS UI in UIKit, it would likely lack features of the usual UI, often be broken or not work correctly, etc. And that's on top of iOS support being generally unstable and broken. Not saying it shouldn't or can't happen - just being realistic about what the result would be so no one is surprised.

I do think using the Android accessibility APIs allows us to use a custom layout, by building an accessibility tree. We don't do this today, but I think we should. This wouldn't help you on iOS, though.

More directly to the setting you referenced, we should really be correctly defaulting that on newer iOS devices. But, this again reflects the state of PPSSPP on iOS. For my part, I have no idea how to detect it correctly.

I know RetroArch already has a custom UI for settings. It's also out of date and broken in multiple ways, but it's possible it might have this particular setting and integrate with VoiceOver. I didn't find a lot of promising results on Google checking if this was true, though. Unfortunately, the last iOS device I had was a 3GS so I have no way to verify.

Maybe it's possible to build a shadow iOS UI on top of the real UI to get the API to work...

-[Unknown]

devinprater commented 5 years ago

I have an idea. Looking through the code of the iOS port, with my 0.2% of coding knowledge, it seems that PPSSPP’s UI can be used by MFI controllers, like iCade and Nimbus and such. If PPSSPP could check to see if VoiceOver is enabled https://developer.apple.com/documentation/uikit/uiaccessibility/1615187-isvoiceoverrunning, then it could speak the item https://developer.apple.com/documentation/avfoundation/avspeechsynthesizer in focus. This could, at the very least, allow games to be selected. Alternatively, PPSSPP could be one of the items in the share sheet https://developer.apple.com/app-extensions/ for .iso or .cso files.

On Dec 30, 2018, at 2:07 PM, Unknown W. Brackets notifications@github.com wrote:

I think it'd be awesome to support this.

It seems like there's no way to integrate with VoiceOver on iOS without either treating things as media closed captions (which doesn't seem like it would work), or entirely using iOS's layout system (UIKit, as you mentioned.) There seems to be support for custom controls https://developer.apple.com/library/archive/documentation/UserExperience/Conceptual/iPhoneAccessibility/Making_Application_Accessible/Making_Application_Accessible.html#//apple_ref/doc/uid/TP40008785-CH102-SW9, but they still have to be part of a iOS dedicated layout.

The unfortunate reality is that PPSSPP is not really "officially" supported on iOS - there aren't many active contributors to the iOS code. Jit doesn't work on latest iOS versions (as you mentioned), it crashes if exited in certain ways, and it's not tested nearly as much. PPSSPP is open source, and not enough iOS users are contributors.

That means even if someone did completely rebuild PPSSPP's iOS UI in UIKit, it would likely lack features of the usual UI, often be broken or not work correctly, etc. And that's on top of iOS support being generally unstable and broken.

I do think using the Android accessibility APIs https://developer.android.com/guide/topics/ui/accessibility/custom-views#populate-events allows us to use a custom layout, by building an accessibility tree. We don't do this today, but I think we should. This wouldn't help you on iOS, though.

More directly to the setting you referenced, we should really be correctly defaulting that on newer iOS devices. But, this again reflects the state of PPSSPP on iOS. For my part, I have no idea how to detect it correctly.

I know RetroArch https://www.retroarch.com/ already has a custom UI for settings. It's also out of date and broken in multiple ways, but it's possible it might have this particular setting and integrate with VoiceOver. I didn't find a lot of promising results on Google checking if this was true, though. Unfortunately, the last iOS device I had was a 3GS so I have no way to verify.

-[Unknown]

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/hrydgard/ppsspp/issues/11696#issuecomment-450583463, or mute the thread https://github.com/notifications/unsubscribe-auth/AOjJzgleSj2KVYdmMiYoGf4JmuqOB5Ozks5u-RzxgaJpZM4ZglUx.

hrydgard commented 3 years ago

Removed the iOS-specific designation for this, since Windows also has an accessibility framework: https://docs.microsoft.com/en-us/windows/win32/winauto/entry-uiauto-win32

It'll probably be possible to create some sort of abstraction that can be accessed from all three accessibility frameworks: Windows, Android and iOS/Mac.

unknownbrackets commented 3 years ago

14202 adds labels to all UI objects (headings, buttons, lists, etc.), although no hookups to note them for accessibility frameworks.

-[Unknown]