Screen reader seems to skip more alerts in phet-io

[stemilymill]

For https://github.com/phetsims/qa/issues/868

Win10 chrome + JAWS

I have noticed that often alerts are skipped after moving the book. More than half the time when testing the standalone phet-io sim, there is no "jiggling more; warmer" and it skips straight to saying that the jiggling and temperature are lessening.

This does happen sometimes in phet brand, but it seems much more frequent in phet-io, both in standalone and launched from studio (even if nothing was changed)

(i am not able to upload the video for some reason)

[Nancy-Salpepi]

For me, with VO in general (PhET and PhET-iO), the alerts that mention the atoms breaking away are usually interrupted.

https://user-images.githubusercontent.com/87318828/210452124-5a4a7fb5-cc15-4d46-8f40-ceed4815ad6e.mov

[zepumph]

I added some code so that the cooling alerts should interrupt the break away alerts way less. I haven't cracked the "left/right" interrupting though, since it is from a different system/module. Can you please test master and see if it sounds any better?

Also I didn't notice any differences between phet-io and phet brands.

[stemilymill]

On chrome + JAWS, I think it is improved in that the alerts for jiggling more and warmer are skipped less often. When they do get skipped, it now seems like they are not being interrupted by the cooling alerts, but by the "A" and "D" when using the WASD keys to move the book. Usually there aren't alerts for left and right when moving the book, but "left" or "right" is sometimes said in between jiggling more; warmer and jiggling less; cooler. @Nancy-Salpepi is VO improved?

[Nancy-Salpepi]

It is a little better. With VO the "left" and "right" statements will still cut off the other alerts. In general it is odd that alerts are getting cut off with VO because they usually don't--they fill the queue and go on forever (not including when assertive alerts are made). On mobile VO I still hear all the alerts.

[zepumph]

So the blend between each of these interrupting each other, cooling/warming/a/d/left/right, is challenging, and imperfect. To me that is secondary to ensuring that you hear when atoms break away, the design is centered around those alerts and that is really important. I know that it is possible to hear a/d/left/right as an interrupt to atoms breaking away, but that also may be correct, since those are directly from a user input. I'm not sure we would want to ignore those, but I could potentially do that though I would want to talk to designers.

[zepumph]

I tried this patch given @Nancy-Salpepi's comment, but it actually didn't help much. What I'm finding with NVDA is that while you have keys pressed, most alerts are not spoken. That makes it challenging to hear the complete picture. I still may commit this though. Next steps for me are to have a bit of a discussion with @jessegreenberg.

Subject: [PATCH] formatting, https://github.com/phetsims/projectile-motion/issues/306
---
Index: js/friction/view/BreakAwayAlerter.js
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/js/friction/view/BreakAwayAlerter.js b/js/friction/view/BreakAwayAlerter.js
--- a/js/friction/view/BreakAwayAlerter.js  (revision 2573038475e1eb29f633a285f123fa9d535d4d73)
+++ b/js/friction/view/BreakAwayAlerter.js  (date 1673027813031)
@@ -15,6 +15,7 @@
 import friction from '../../friction.js';
 import FrictionStrings from '../../FrictionStrings.js';
 import FrictionModel from '../model/FrictionModel.js';
+import AriaLiveAnnouncer from '../../../../utterance-queue/js/AriaLiveAnnouncer.js';

 // constants
 const capitalizedVeryHotString = FrictionStrings.a11y.temperature.capitalizedVeryHot;
@@ -57,7 +58,10 @@
     // @private
     this.utterance = new Utterance( {
       alert: new ResponsePacket(),
-      priority: Utterance.HIGH_PRIORITY
+      priority: Utterance.HIGH_PRIORITY,
+      announcerOptions: {
+        ariaLivePriority: AriaLiveAnnouncer.AriaLive.ASSERTIVE
+      }
     } );

     // @private

[Nancy-Salpepi]

I had mentioned a while ago that I was missing A LOT with NVDA because it doesn't speak any alerts until you stop pressing a key. Definitely easy to miss things. https://github.com/phetsims/friction/issues/296

[zepumph]

Right! Thanks for that. We closed that likely because there is nothing to be done. It is good to note https://github.com/phetsims/friction/issues/296#issuecomment-1152345163 over here though. Thanks @Nancy-Salpepi

[zepumph]

Screen reader seems to skip more alerts in phet-io

@jessegreenberg mentioned his best guess here would be about performance. I tested with ?profiler in https://phet-dev.colorado.edu/html/friction/1.6.0-dev.28/phet-io/friction_all_phet-io.html?postMessageOnError&phetioStandalone&profiler and it was running around 40FPS, then I turned on NVDA and it immediately went to 15 FPS. Weird! I will need to do some more research.

UPDATE: Ahhh, false alarm, it was because I turned on the NVDA highlighting. Turning that off fixed things.

@jessegreenberg and I really like https://github.com/phetsims/friction/commit/ffe597de001a452521fbcd691d4028c8afa55b73.

In general we feel like we have always felt: we wish we could do more to control the ordering of aria-live alerts. All things considered though, the two commits in this issue really help. I'll take a look at the PhET-iO stuff.

[zepumph]

It has taken me far too long to realize this, but over in https://github.com/phetsims/friction/issues/321 we fixed the fact that the book dragging had become REALLY slow. @stemilymill and @Nancy-Salpepi could you please spend a bit of time retesting the cases that brought you to comment here and see if things got a lot better with faster dragging. The main piece with this is that you have to change direction much less frequently, so "a/d/left/right" interrupt important alerts much less frequently.

[Nancy-Salpepi]

I think things got a bit worse with the faster dragging using VO.

https://user-images.githubusercontent.com/87318828/211360448-16b755d8-22ee-49ec-8057-58d8e184febe.mov

[Nancy-Salpepi]

@stemilymill how does it look on your end?

[zepumph]

Thanks @Nancy-Salpepi. Can you please also compare your experience to the published version? Nothing should have changed too much in the design of description since that, but it is quite possible that timing issues have snuck in. If it is just as bad, @emily-phet and I think that we may want to punt.

[Nancy-Salpepi]

@zepumph Something has definitely changed with VO. The alerts used to pile up and now they get interrupted too much! In the published version I heard the "more atoms break away" statement 2/17 times. 🏈

[terracoda]

@zepumph, re https://github.com/phetsims/friction/issues/319#issuecomment-1372924749, I am not sure what you need to crack, but if the left-right responses are getting in the way, they are of the least importance. Their main function is to let blind keyboard users know that they are moving but not creating friction. Ideally, they are only heard when there is air space between the books, and even then, that space may not last long, so if they are interrupted that is completely fine.

[jessegreenberg]

In https://github.com/phetsims/friction/issues/319 @Nancy-Salpepi reported the problem happens in Chrome as well, so that indicates in an issue with VoiceOver and not Safari.

[Nancy-Salpepi]

In https://github.com/phetsims/qa/issues/886, I tried to use short key strokes with VoiceOver. 50% of the time I still didn't hear the entire statement "Very Hot. More atoms break away." It was either interrupted or entirely skipped over.

[zepumph]

I believe that this is actually the same issue as https://github.com/phetsims/friction/issues/329. We had places in the code where we were clearing out alerts before they were being heard. @Nancy-Salpepi can you please double check this issue and comment on what else there is to do and we will got ahead and cherry pick the commits above to the RC branches.

[Nancy-Salpepi]

Sorry @zepumph. The break away alerts still sometimes get interrupted with VoiceOver.

I noticed that if an entire row breaks off, I hear the entire alert. If only some atoms break away, the alert gets interrupted. Here is a video showing both cases:

https://user-images.githubusercontent.com/87318828/216133200-01a8a28f-b895-4ab6-8b17-a4aa59ddb637.MOV

[zepumph]

@jessegreenberg, looking at the above video, should we get rid of the left/right alerts? Or perhaps some of them?

[zepumph]

@Nancy-Salpepi how about that on master? Now the left/right shouldn't interrupt other more important alerts as much. But since it is description, we don't have a ton of control on our side before it is sent to aria-live.

[jessegreenberg]

The video isn't loading for me for some reason. But I don't know if we can do too much about this considering https://github.com/phetsims/a11y-research/issues/182.

[Nancy-Salpepi]

When left/right alerts occur seems very random to me. Sometimes I hear them too much and sometimes rarely. They can 'interrupt jiggling more/less warmer/cooler" statements at times, but as you said @zepumph, not all the time.

[zepumph]

I don't really know how much more effort to put here, I also feel like our changes are going to continue to step more and more on the original design decisions, and potentially cause other regressions.

My general sentiment about this issue is, "its a bummer that we don't hear some items, and sometimes it is important items." Is that an appropriate summary about the voice over experience?

I wonder if we should just proceed as is.

[jessegreenberg]

That is my recommendation and preference. If Apple is able to fix https://github.com/phetsims/a11y-research/issues/182 in the future we might have more confidence about making changes to improve this.

[zepumph]

Excellent. Thanks everyone for diving in here and taking it on. Marking as ready for some cherry picking.

[jessegreenberg]

I am a little confused because this issue refers to dev.28, but maybe some fixes here happened after the rc.1 was built.

Commits referenced in this issue are: https://github.com/phetsims/friction/commit/2a93dfd96d9b15885c7a30ef9412445a9e2430b8 https://github.com/phetsims/friction/commit/ffe597de001a452521fbcd691d4028c8afa55b73 https://github.com/phetsims/friction/commit/945a1fb53b4a593505ade5d7c92ee531d1632ab5

The first two are in the history of friction-1.6. So the last one is the only one that needs to be cherry-picked.

[jessegreenberg]

Done, ready to verify.

[Nancy-Salpepi]

In rc.3, description is definitely better than in was in the beginning in testing. I can hear getting warmer and break away alerts now, although they are still interrupted at times. Closing.