Voicing does not work with Chromebook

Nancy-Salpepi commented 1 year ago

Test device Chromebook

Operating System 109.0.5414.94

Browser chrome

Problem description For https://github.com/phetsims/qa/issues/886, Voicing doesn't work for Friction using the Chromebook. When I click on things I see the little megaphone appear in the sim tab, but I don't hear anything. Voicing only seems to be working in published GFLB and JT (not in published RaP). It didn't work for any of the sims I tried in master (Friction, RaP, GFLB).

Voicing works with mac + chrome and win10 + chrome.

Troubleshooting information: !!!!! DO NOT EDIT !!!!! Name: ‪Friction‬ URL: https://phet-dev.colorado.edu/html/friction/1.6.0-rc.1/phet/friction_all_phet.html Version: 1.6.0-rc.1 2023-01-17 16:32:22 UTC Features missing: applicationcache, applicationcache, touch Flags: pixelRatioScaling User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36 Language: en-US Window: 1536x781 Pixel Ratio: 1.7999999523162842/1 WebGL: WebGL 1.0 (OpenGL ES 2.0 Chromium) GLSL: WebGL GLSL ES 1.0 (OpenGL ES GLSL ES 1.0 Chromium) Vendor: WebKit (WebKit WebGL) Vertex: attribs: 16 varying: 31 uniform: 1024 Texture: size: 16384 imageUnits: 16 (vertex: 16, combined: 32) Max viewport: 16384x16384 OES_texture_float: true Dependencies JSON: {}

jessegreenberg commented 1 year ago

I have a chromebook, I can take a look.

jessegreenberg commented 1 year ago

I just tried 1.6.0-rc.1 on a chromebook 104.0.5112.83 (Official Build) (64 bit) and heard Voicing come through. Then I upgraded to 105.0.5195.134 (Official Build) (64 bit) and still heard Voicing come through. Then I upgraded to 109.0.5414.94 (Official Build) (64 bit) and all Voicing was broken. Something about Chromebook changed.

EDIT: I am seeing that Voicing on latest ChromeOS works ~5% of the time. Usually I hear nothing but sometimes I do hear "Voicing On" or the content from a reading block.

EDIT: I confirmed it does seem better in published JT. Though there was once or twice where I didn't hear any content.

EDIT: When we do not hear anything, window.speechSynthesis.speaking is stuck true. I tried to call speechSynthesis.cancel and it did not change the value of speaking. Exact same behavior as https://github.com/phetsims/a11y-research/issues/183.

EDIT: Here is the output from using the debug option of UtteranceQueue

Not ready to announce because synth is stuck "speaking".

I made this demo HTML:

<!DOCTYPE html>
<html>
<head>
  <meta charset="UTF-8">
  <title>TEST PAGE</title>
</head>
<body>
  <button id="button">Speak!</button>
</body>

<script>

  const button = document.getElementById( 'button' );
  button.addEventListener( 'click', event => {
    speechSynthesis.speak( new SpeechSynthesisUtterance( 'Hey there' ) );
  } );

</script>
</html>

At first I didn't hear anything. Then I closed all other sim tabs and I started to hear the "Hey there" every button press.

jessegreenberg commented 1 year ago

I found that if I remove the workaround added in https://github.com/phetsims/gravity-force-lab-basics/issues/303 the problem goes away. The problem from that issue does come back though.

Specifically: https://github.com/phetsims/utterance-queue/blob/2f64e9580771087750bdc3f1c5ac778abb089f1f/js/SpeechSynthesisAnnouncer.ts#L448-L457

I believe that the times Voicing was working on Chrome 109 were the times I started to use it before the workaround was used.

I was curious why this issue is NOT in published GFL:B, even the workaround references an issue in that sim. But it is explained by this comment: https://github.com/phetsims/gravity-force-lab-basics/issues/303#issuecomment-952408265

I tried putting a string in the workaround like

synth.speak( new SpeechSynthesisUtterance( 'waking' ) );

and the problem was fixed. Then, I tried speaking an empty space and the problem still seems fixed. Yuck...

The issue is that speaking an empty string breaks SpeechSynthesis in ChromeOS. I verfied this by changing my HTML test to

<!DOCTYPE html>
<html>
<head>
  <meta charset="UTF-8">
  <title>TEST PAGE</title>
</head>
<body>
  <button id="button">Speak!</button>
</body>

<script>

  const button = document.getElementById( 'button' );
  button.addEventListener( 'click', event => {
    speechSynthesis.speak( new SpeechSynthesisUtterance( '' ) );
  } );

  window.setInterval( () => {
    console.log( speechSynthesis.speaking );
  } )

</script>
</html>

And when I press the button I see Screenshot 2023-01-27 4 26 03 PM

So that is indeed the problem.

@zepumph can we meet next week to discuss whether we should continue with this workaround (adding a space), possible maintenance releases, or removing this workaround entirely to avoid workaround maintenance pain.

jessegreenberg commented 1 year ago

Discussed. We prefer adding a space to the workaround. It seems like the least amount of effort, and lets us keep using the workaround for slow speech synthesis on Chromebooks.

Back to me to add the space. Ill look at which published sims have this problem and use that to inform whether we proceed with a maintenance release.

zepumph commented 1 year ago

@jessegreenberg, I'm going to mark this as ready to cherry-pick since the original change for friction has been made, to be verified in RC and/or with you on master. Thanks!

jessegreenberg commented 1 year ago

@Nancy-Salpepi can you please verify that this is fixed in master?

Nancy-Salpepi commented 1 year ago

Great job! Working just fine in master with Friction (and RaP).

jessegreenberg commented 1 year ago

I think that is it for this issue, the only published sim with this problem is RaP (https://github.com/phetsims/ratio-and-proportion/issues/550).

Commit to cherry pick: https://github.com/phetsims/utterance-queue/commit/feff87d79a77545af1ea7219521494a3ce2d123a

jessegreenberg commented 1 year ago

cherry-picked into the friction-1.6 branch of utterance-queue. Ready to verify.

Nancy-Salpepi commented 1 year ago

This looks good in rc.3. Closing.

phetsims / friction

Voicing does not work with Chromebook #328