ARIA with MathML 3 examples?

[dginev]

Hi everyone,

Work-in-progress issue here, hoping to collect feedback and links to related work.

The newly restarted Math WG is gradually making headway with reevaluating MathML accessibility. Especially so in order to allow accessible navigation through a larger math expression, by incrementally annotating the MathML tree.

I experimented with a few smallish examples, using the existing ARIA and MathML 3 recommendations, and it seems more promising than I anticipated:

https://hackmd.io/@dginev/SkBHsZTiO

At least when voiced with Chrome and Screen Reader, the three examples yield:

1. narration bionomial a b for aria-labelledby="op arg1 arg2"
2. narration point x at coordinates 0 0 for aria-labelledby="point varname at-coordinates xy"
   • Clicking and staying focused on "X" yields the narration "X, unit circle origin", for aria-describedby="origin-details"
3. narration x is in open interval 0 1 for aria-labelledby="varname in interval"

with the respective ids pointing into MathML subexpressions, or - for the aria-describedby ids - to HTML spans.

Neatly, this allows reading the pieces in isolation (e.g. just the interval mrow element), without any extra guess work or inference -- the narration is fully encoded in the ARIA annotations. In other words, we can do navigation immediately.

Such an approach also addresses one criticism in #660 that

Presentation MathML is too limited to express many mathematical concepts (e.g,. simple commutative diagrams). It is also limited by focusing on single expressions without (reasonable) integration into larger document structures which causes issues in many basic educational settings (e.g., explanatory diagrams in elementary school books are usually impossible).

While it is true that MathML only focuses on standalone math expressions, the ARIA scheme of composing lists of document-level id attributes allows to combine say MathML and SVG element ids from an outer HTML element with an aria-labelledby attribute.

We are yet to discuss these examples in more detail in our Math WG meetings, and our participants do not necessarily include any ARIA experts. I think anyone interested who is reading here would be extremely welcome to join us and share an ARIA perspective / listen-in on what we're upto.

To end on a specific question: Are there additional ARIA+MathML writeups, particularly with rich examples, that we should consider in our explorations? It would be very helpful if you could point me to some resources that come to mind. Thanks!

[cookiecrook]

Related: Some examples of how to use VoiceOver natively with MathML captured here: https://www.perkinselearning.org/technology/blog/how-read-math-expressions-voiceover-ios-device

Of note, the video creator does not cover the full screen exploration (including touch exploration), or the similar Mac experience. The video creator does demo the Nemeth braille support on those... which is also available on Mac as well as iOS, and in WebKit/Safari, not just in the Books app experience that is demoed.

[cookiecrook]

IMO, using ARIA labelledby to override the native behavior (as in the example above) would result in a worse experience... MathML (and its accessibility) was removed from Chromium/Blink 5+ years ago, so I think this is just a browser bug.

Joanie also mentioned on the call that Igalia was currently re-implementing MathML in Chromium. So they should be able to provide the braille and sub-formula exploration features available on Mac, and should be able to better support MathML on Windows too. The Windows braille support may or may not require additional work from Windows AT developers.

[jnurthen]

@NSoiffer can you help on this?

[cookiecrook]

I seem to have deleted a comment. 👎 IIRC, it was something like:

iOS and Mac have had native MathML support for since 2014 or so. I think more people aren't aware of it because most of the popular Math polyfills remove the original MathML support, which in turn removes VoiceOver's ability to support the formula exploration and Nemeth braille support.

[NSoiffer]

There seems to be two separate things currently being discussed in the thread:

1. @dginev 's suggestion for using aria to get speech for math
2. MathML support in browsers

I'll start with the support issue first since it is easiest: Igalia has been working on getting MathML support into Chrome (and hence Edge, etc). It is maybe 80% of the way there and is behind the "experimental features" flag. Getting the support into Chrome only means that MathML is displayed -- it does nothing for accessibility. In fact, if you are using Chrome and a screen reader (VoiceOver, JAWS, NVDA, and (likely) Orca [I haven't tried it]), you will have a better experience with the MathML than a sighted person because they all work with MathML now. Because of ambiguities and the need for heuristics to get a semantic reading, you will get better or worse readings from each screen reader. E.g, with the first example Deyan has, NVDA+MathPlayer will say "a choose b" and JAWS will say "fraction a over b" (which is not correct).

FYI: MathJax moves any MathML into a hidden display (0x0px area) so screen readers see it but sighted users don't and also sets aria-hidden on the displayed part so screen readers don't see that. It's a nice tradeoff, but not perfect (e.g, you can't easily get visual feedback while navigating the math).

Onto the what Deyan suggests... Neither JAWS nor NVDA use aria labels when reading MathML. That's likely wrong and should be called out so that screen readers make use of them (since I wrote MathPlayer which is used by NVDA, I should fix that...). I downloaded Google's screen reader and it too ignored the ARIA labels (and does a poor job of reading the math example).

One problem with Deyan's idea is that math has its own braille codes. By using ARIA labels, if screen readers used them to override builtin semantics, the braille would be wrong. aria-braillelabel solves this problem (I think math was a prime motivator for it), but that now puts the onus on the generator of MathML that generated the area-label to also generate the appropriate braille. That might be a technical challenge. Perhaps a bigger problem is that in English countries, there are now two competing braille math codes: UEB (math) and Nemeth [UEB math vs Nemeth is controversial, but that's not something the W3C or authors can do anything about]. It would be best for the braille to be generated when the user preference is known. Perhaps AT can fill in the braille if it sees a aria-label in math. If that is the intention, perhaps the ARIA 1.3 draft section about braille labels should call that out.

Another problem with using labels is pronunciation of the letter "a". "a" can have a short or long sound, but in math, it is always a long "a" sound. NVDA gets it right because it forces the pronunciation; JAWS doesn't and I've seen lots of complaints about it because it tends to make the math unintelligible. A work around is for screen readers to check if the label is "a" and in that case, don't send that text directly but instead force the pronunciation.

Deyan has only recently come up with this approach and it has not been discussed at a MathML WG meeting. I'd also like to hear from ARIA experts what they think about his approach and what suggestions people have to deal with the braille and forced pronunciations issues I mention.

[dginev]

Hi @cookiecrook and @NSoiffer ,

I appreciate the eagerness to discuss high level strategy, history and pragmatics of approaches to MathML. I think there are better venues for those discussions than this particular github issue however (the Math WG mailing list for one). I opened the thread here with the only intention to collect existing examples of ARIA used with MathML, which the ARIA group may be maintaining / be aware of.

If such a collection does not exist, that is perfectly fine, and I'll accept that as a resolution. Feel free to close the issue if I have hit on uncharted territory.

My examples were to illustrate what flavors of annotations I am looking for, and as mentioned above were experiments I found promising enough as to come here and check if there are other resources out there to compare against / adjust towards. My goal here wasn't to file any specific proposals, and as Neil mentioned the examples are to undergo discussions in the recent future.

As to the link to:

https://www.perkinselearning.org/technology/blog/how-read-math-expressions-voiceover-ios-device

Thanks, that seems to cover the "baseline" presentation MathML without ARIA. That clearly is a healthy starting point for accessibility (compared to say PNG), but I'm fishing for examples using ARIA itself, which could be leveraged to produce fluent narrations with the annotating hand of an author.