Mobile VO skips content in the PDOM

[jessegreenberg]

Identified in https://github.com/phetsims/friction/issues/292 https://github.com/phetsims/greenhouse-effect/issues/195 https://github.com/phetsims/john-travoltage/issues/449 https://github.com/phetsims/friction/issues/308

Mobile VO sometimes skips PDOM content.

---

Notes from testing:

In the case of friction, after particles the second two paragraphs in the screen summary are no longer readable in the PDOM.

I thought it might have something to with aria-live, but it doesn't happen after pressing the ResetAllButton which triggers an alert. I tried changing the ResetAllButton alert to assertive but it made no difference.

In one test, I found that content disappeared in friction only after all particles broke away from the book.

I noticed that in Friction, the content that gets skipped is nested under a div:


I tried pulling the paragraphs out of that div, it did not fix it.

I noticed turning VO on and off again fixes the problem.

I removed the div mentioned in the image above. VoiceOver still skips the first paragraph that was under the div, but does read the second one.

I noticed that the paragraphs that gets skipped are the ones whose content changes.

Ive been creating a more and more complicated test case, trying to reproduce what we are seeing in the sim. The demo as an aria-live element that spams alerts and paragraphs of text content that update. Ive tried

• Keeping static paragraph content
• Updating paragraph innerHTML and innerText
• Removing then adding new paragraphs with content to the DOM
• Using CSS to have the PDOM styled the way we have it in the sim
• Using polite and assertive alerts
• Increasing/decreasing the rate of alerts.
• Removing then adding the same paragraphs with content to the DOM

So far I haven't seen the bug. Heres the html

```html

My Heading

Friction is an interactive sim. It changes as you play with it. It has a Play Area and a Control Area.

Chemistry book has far fewer jiggling atoms as many have broken away. Chemistry book rests lightly on top of a Physics book, and is ready to be rubbed against it. In zoomed-in view of where books meet, surface temperature is cool, and atoms jiggle a tiny bit.

Reset sim to make more observations.

If needed, check out keyboard shortcuts under Sim Resources.

Toggle alerting ```

I have tethered my ipad to my Mac and I can maniuplate the Friction PDOM in real time while VoiceOver runs. Ive been changing tags and removing attributes to see if anything impacts things. So far nothing has made a difference.

I was changing attributes and innerHTML wildly without any changes like this:

You can see the paragraph that is not being read highlighted in the DOM. Also some new paratraphs that I added to the DOM, VoiceOver is ignoring those too. But then I toggled "All Audio" off and suddenly VoiceOver is reading all the content again! But maybe that is because the sim crashed, I got a "cannot read null" error in the dev tools. No, I verified that turning off all audio makes issue go away. Why?

Once VoiceOver starts reading the paragraphs again, it is more difficult for it to break again but it does happen eventually. I removed all particles from the book ~3 times before the screen summary content became undescoverable again.

I have noticed a couple of times in the sim that the paragraph becomes undiscoverable when reading it gets interrupted by an assertive alert. I tried but have not been able to observe this in my vanilla HTML.

I tried removing the work of the "All Audio" toggle button and its aria-pressed state and still found that pressing that button makes the issue go away.

I found that if I wait for all alerts to be spoken (waiting until the "reset sim to make observations" alert in Friction) I do not see this bug. I think that is an important trigger for the bug.

I tried disabling sound in case that was causing this problem because pressing the audio button seems to fix this. But the behavior is the same. For some reason pressing the "All Audio" button makes the problem go away, even when the button does nothing.

I am now seeing that pressing the "Reset All" button makes the problem go away. Is it an activation event that fixes it? I just produced the problem, then double tapped on a book to pick it up and release it. It made the problem go away.

I disabled all alerts and I still see the problem. So it has nothing to do with aria-live.

I tried forcing a re-render of the DOM element in the PDOM by toggling it back and forth between a div and a p. No impact.

I tried in my example HTML changing the DOM content with very long strings in the paragraph content (~8 seconds to read) like in the sim. No impact.

I tried in my example HTML having the paragraph content update while I doing a custom drag and drop action. I pressed the "Toggle Alerting" button and then double-tap and held the button to start drag/drop events while the paragraph text content updated. No impact.

```html

My Heading

Friction is an interactive sim. It changes as you play with it. It has a Play Area and a Control Area.

Chemistry book has far fewer jiggling atoms as many have broken away. Chemistry book rests lightly on top of a Physics book, and is ready to be rubbed against it. In zoomed-in view of where books meet, surface temperature is cool, and atoms jiggle a tiny bit.

Reset sim to make more observations.

If needed, check out keyboard shortcuts under Sim Resources.

Toggle alerting ```

I tried testing with a full copy of the PDOM in Friction, and adding a script to update the content of the summary paragraph every 5 seconds. I tested with drag/drop input. The bug did not appear.

```html

Friction

Friction is an interactive sim. It changes as you play with it. It has a Play Area and a Control Area.

Chemistry book has far fewer jiggling atoms as many have broken away. Chemistry book rests lightly on top of a Physics book, and is ready to be rubbed against it. In zoomed-in view of where books meet, surface temperature is cool, and atoms jiggle a tiny bit.

Reset sim to make more observations.

If needed, check out keyboard shortcuts under Sim Resources.

Play Area

Grab Chemistry book

Look for grab buttons. Once grabbed, use keyboard shortcuts to move book or zoomed-in book.

Grab zoomed-in Chemistry book

Control Area

Reset All

Sim Resources

Preferences All Audio Keyboard Shortcuts PhET Menu

```

[jessegreenberg]

I have exhausted just about everything I can think of here and ready to recommend we list this as one of the known AT bugs.

[jessegreenberg]

@terracoda suggested running the HTML through a validator to see if it reports anything amiss.

EDIT: I ran the validator, no errors. Just a few warnings that seem unrelated.


EDIT: "View page source" gave me incomplete HTML, copying HTML from the dev tools showed me these errors from the validator". HTML copied was when the PDOM is in the state that VO skips content:


Both are attributes on the "Sim Voicing" button in the Toolbar, very unlikely to be causing this.

@terracoda also suggested that maybe unicode marks are causing this. We have seen unicode interfere with AT output in the past. I looked for unicode or other artifacts in the skipped content and didn't see any. My understanding is that unicode marks are added in built versions only and we are seeing this in both built and unbuilt versions

[jessegreenberg]

@terracoda recommended testing without any CSS, that was a good idea. Here is a test case that I set up by 1) Disabling all PDOM CSS. 2) Disabling all PDOM element positioning. 3) Using the sim to put it in the VO broken state. 4) Tethering my iPad to my Mac and removing all graphical elements with dev tools. 5) Reading through paragraph content with VoiceOver.

The problem STILL happens even when there is only the unstyled PDOM left.

Here is the result:

https://user-images.githubusercontent.com/6396244/181793164-bac58ab9-0aa8-44da-a3ff-bc0d48339eee.mp4

Once graphics are removed, I swipe with VoiceOver to try to read all paragraphs. You can see content skipped. Near the end of the video I tap on the skipped paragraph with my finger and that makes the problem go away.

This could get us closer to identifying what is causing the issue if we want to continue.

[jessegreenberg]

@zepumph and I met to try to isolate the cause of this. We returned to the example HTML pages and found that this IS happening outside of PhET sims. I don't know why I wasn't seeing it when testing alone, maybe I failed to clear caches, or HTML updates happened to slowly.

Anyway, here is the most simple breaking case:

```html

My Heading

Friction is an interactive sim. It changes as you play with it. It has a Play Area and a Control Area.

Summary HERE!

Reset sim to make more observations.

If needed, check out keyboard shortcuts under Sim Resources.

```

We found that paragraph will become undiscoverable with iOS VoiceOver whenever its textContent changes. Maybe VoiceOver assumes that paragraphs are static? That doesn't explain why paragraphs become totally undiscoverable. But we though attributes role='status' aria-live="off" were the most correct way to tell the AT that the paragraph is going to change. But adding this did not fix the problem.

But we did find a workaround that DOES fix it: adding role="text" to the paragraph element. We got the idea from https://stackoverflow.com/questions/23526161/for-ios-safari-voiceover-how-do-i-get-voiceover-to-read-something-other-than.

text is not even a real role, it is not clear why this works. It feels very hacky. And we would need to build setting of this attribute into scenery/ParallelDOM.ts somehow so it is set for all paragraphs.

[jessegreenberg]

We wanted to run our findings by @terracoda and @emily-phet to help decide where to go from here. We did find a workaround but do not recommend using it. This is clearly an iOS problem and the workaround is totally hacky. It seems to work now but since it is outside of any DOM specification it could also stop working any time. If we proceed with it, we will do work in scenery to make sure that that all p elements get a role="text" attribute when they are rendered. Do you have any thoughts about this?

[emily-phet]

@jessegreenberg I think we shouldn't implement the workaround - but great that's it's more clear what the root of the issue is.

[zepumph]

But we though attributes role='status' aria-live="off" were the most correct way to tell the AT that the paragraph is going to change. But adding this did not fix the problem.

I think it would be nice to have a non-blocking conversation about adding these to our screen summary paragraphs. It would be a nice improvement in my mind to convey that this is not just a paragraph, but a dynamic text space that updates with the sim's updates. I defer to the A11y designers though.

[terracoda]

Excellent detective work. @jessegreenberg, you won’t believe this, but I almost posted that the changing content could be the reason in slack. This was the first sim where I thought it would be nice to trace/track the rising and falling results of making or not making friction, even though it is difficult to catch live while changing.

This might also be an issue for MAL’s observation window, and GHE might have a dynamic Scene Summary similar in structure to the summary in Friction.

There are a couple of design things we should seriously discuss and consider for the long run:

1. Ask, is describing a changed state (clarification: in an actual State Description) as it happens super fast essential for learning? (At design time I thought it was, maybe it is not).
2. Can the change be captured differently. For example, in John Travoltage's scene summary, we describe the discharge event in the past tense and note the change in the amount of charge.
3. And I have one more technical question…Do list structures work any differently than paragraph structures?

I agree with @emily-phet that we should not implement the hack. It’s not a long term solution.

I do feel not having access to the scene summary in Friction - its single dynamic state description - is a bit of a problem which I think can be resolved with a small amount of re-wording and a reduction in parameters- the temperature is either dropping or already at cool.

In retrospect, apologies for designing such accurate state descriptions that screen reader software can’t handle them 😀

At least not as of July 2022.

[jessegreenberg]

Ok, thanks @emily-phet and @terracoda! I'll write a bug report to Apple about this just in case.

I'll also add this to the list of known AT problems.

[jessegreenberg]

Bug report submitted to apple, we will see if they respond.

[jessegreenberg]

@terracoda shared that there is a special address that Apple has for critical accessibility problems, Ill try and submit this issue there as well.

[terracoda]

I am running iOS 15.7.8 on my iPhone 7 and I still seem to experience this issue.

I hear the description for the Chemistry book on sim load, but then after interaction I can't seem to find that description with horizontal or vertical swipes using iOS VoiceOver.

[terracoda]

I can however grab and rub the books, so the custom grab is now working for Friction. What issue is related to that bug?