dylan-fox
commented
3 years ago Consider examining Last of Us Part II as an example of using sound to guide users through an interactive 3D environment. Naughty Dog added an optional "Enhanced Listen Mode" as part of their vision accessibility features that includes unique audio cues and text to speech. (Also has amazing contrast enhancements and other features as well.)
Summary of Last of Us II accessibility features
Text-to-Speech and Audio Cues
Any information that is only represented visually is unavailable to a blind player. To address this discrepancy, we created several options to convey this information through sound and controller vibration instead. Traversal and Combat Audio Cues work alongside the game’s audio to map easily identifiable sounds to commonly used actions (scavenging pickups, jumping across gaps, dodging incoming melee attacks, etc.). These audio cues are catalogued in an audio glossary that you can refer back to at any time. Additionally, our menus and other on-screen text can be voiced using the Text-To-Speech option. For those that have their language set to English and Text-to-Speech enabled at the console level, The Last of Us Part II will automatically enable Text-to-Speech for you in-game.
Individual features
Text to Speech Enables narration of on-screen text. When enabled, request a text-to-speech description of your current status with |touchpad-swipe-up|. Text-to-Speech volume can be adjusted from the Audio Volumes menu.
Traversal Cues Enables additional accessibility audio cues to assist with traversal and exploration. Plays a sound cue for: • Jumpable gaps, climbable ledges, and squeeze-throughs • Areas that require crouching or going prone• Interacts and pick-ups • Nearby breakable glass • Gaining or losing movement control • When Ledge Guard is enabled, plays a sound cue when it activates For additional sound cue information, consult the Audio Cue Glossary.
Combat Audio Cues Enables additional accessibility audio cues to assist with combat and stealth. Plays a sound cue for: • Enemy grab and strike prompts • Incoming enemy melee attacks • Being impaled with an arrow • Aiming at an enemy • Landing a shot on an enemy For additional sound cue information, consult the Audio Cue Glossary.
Combat Vibration Cues Enables vibration cues to assist with combat. Plays a vibration cue for: • Incoming enemy melee attacks • Aiming at an enemy • Landing a shot on an enemy
Guitar Vibration Cues Enables a vibration cue when the correct guitar note for story progression is selected.
This is what a framework that is trying to do A11y in WebGL looks like: https://github.com/pmndrs/react-three-a11y
What can we do as an API to take the baton from where they are and allow WebXR scenes to hook into real assistive technology, such as screen readers?
Telling developers to use an API like Speech Synthesis to provide audio based assistance isn't very useful because there is a world of difference between how speech synthesis screen readers behave & sound. Users that rely on speech based assistive technologies will have it configured to their preferences and it usually sounds very different, usually a lot faster, compared to basic synthesised speech.