phetsims / utterance-queue

Alerting library powered by aria-live
MIT License
0 stars 2 forks source link

Mimic the SpeechSynthesis API for Android #63

Closed jessegreenberg closed 2 years ago

jessegreenberg commented 2 years ago

This is a specific need of https://github.com/phetsims/fenster/. Fenster is a Cordova app running a PhET simulation. When running on Android, the app runs the simulation in an Android WebView. window.SpeechSynthesis is not supported in Android WebViews so we need to make requests to the Android Text to Speech engine instead. This will presumably happen by making requests with window.postMessage from the simulation to be received by app code, which then makes speech requests with Android's Text to Speech solution.

The Voicing features are only enabled if SpeechSynthesisAnnouncer.isSpeechSynthesisSupported returns true, which checks for the existence of window.SpeechSynthesis. So in order to support speech synthesis in this environment we need to

1) Create an implementation of window.SpeechSynthesis that forwards requests to speak to the Android app with postMessage. 2) Assign the implementation of SpeechSynthesis to the window in this environment. For this strategy to work it must be assigned before the call to SpeechSynthesisAnnouncer.initialize(), and before other Voicing features are initialized. 3) SpeechSynthesisAnnouncer is very dependent on whether the synth is still speaking. Hopefully we can implement this in our implementation by sending information back to the sim from the Cordova app.

We considered implementing this directly in Fenster and assigning SpeechSynthesis to the simulation window from outside the WebView. But it seems that when we set src on the iframe element (in Cordova) the window variables change. And we cannot set things to the window after the simulation has loaded because by then the sim will have tried to initialize Voicing and found that SpeechSynthesis is not available, thus disabling the feature. So we think the best way to do this is to assign a SpeechSynthesis implementation within the simulation code.

jessegreenberg commented 2 years ago

@zepumph helped me brainstorm this. He found a way to assign something to window.speechSynthesis before the simulation starts to load from the Fenster app, so we are going to proceed with a Fenster specific solution there.

We discussed that it could be nice to have a general SpeechSynthesis "polyfill" solution in utterance-queue. But the implementation of this will be specific to fenster. And the only place this will be needed is on the Android WebView platform anyway. Closing, will reopen a new issue in Fenster.