search

mathjax / MathJax

Beautiful and accessible math in all browsers
http://www.mathjax.org/
Apache License 2.0
10.07k stars 1.16k forks source link

Expose MathML for accessibility #938

Closed jcsteh closed 8 years ago

jcsteh commented 9 years ago

In order for screen readers and other assistive technology (AT) products to provide rich access to math content, they need to be able to access the raw math in a standard way. Providing alternate text is not sufficient, since although this does allow for better spoken math, it does not provide for braille, interactive exploration, etc. It seems MathML has become the dominant standard for exposing math content and MathJax uses MathML internally, so it'd make sense to expose it as MathML.

You can already use the toMathML extension to access MathML for a MathJax tree. The problem is that this requires JavaScript calls and an AT does not always have the ability to execute arbitrary JavaScript on the page. (For example, in Firefox, this isn't possible without a browser extension or Greasemonkey script.) There needs to be some way to obtain MathML from MathJax that does not require JavaScript calls.

One solution is to expose the MathML via an attribute; e.g. data-mathml. This is pretty inelegant and non-standard, but it does solve the problem. I've already implemented experimental support for this in NVDA's (not yet released) math support.

If exposing an attribute like this automatically is considered too expensive, we might be able to come up with a solution using an off-screen button or similar which exposes the attribute when activated. However, this is even less elegant and is more error prone, so I'd hope to avoid something like this if possible.

/cc @pkra

sinabahram commented 9 years ago

Is there a page to check out the menu for keyboard and other accessibility in the form that you’re thinking of implementing it?

jcsteh commented 9 years ago

Would it be possible for an AT to set a flag in the MathJax cookie so that the AT's users wouldn't have to opt-in for accessibility?

I don't think that's possible in browsers where we can't call JavaScript, which basically means anything other than Internet Explorer.

What about the MathJax menu btw? James said that you can not route the mouse to some "hidden" math, but for the special technique of 1px by 1px (which is not entirely hidden) it might be possible to do so.

That might work for the menu, as long as right clicking the pixel occupied by the fake (1px by 1px) native MathML activates the menu.

pkra commented 9 years ago

Thanks again for providing more input.

Here's what the next steps are from the MathJax team (fyi that's @dpvc, @zorkow, @christianp, @pkra).

We are temporarily locking this thread to reduce noise and frustration on all sides. We will unlock the issue once we get back to working on these items (fyi, this issue remains in the milestone for the next release but the extension could be cherry picked as soon as it's done). Feel free to post on mathjax-dev@googlegroups.com for general discussions or reach out to me personally.

pkra commented 9 years ago

Some preliminary test results for the AssistiveMML.js extension.

pkra commented 9 years ago

PS: I've unlocked the issue.

dpvc commented 9 years ago

The extension is available in the assistive-mml branch of my fork of MathJax. It has not yet been merged to the develop branch. The details of what aria labels should be used where still needs to be settled.

pkra commented 9 years ago

For the record, that branch is https://github.com/dpvc/MathJax/tree/assistive-mml

fsrg commented 9 years ago

Peter,

Have you run any tests with JAWS? Let me know if you need assistance with that.

Thanks, RG

pkra commented 9 years ago

@fsrg I'm afraid I don't have access to JAWS. It would be great to hear if things work there.

If you do, here's a live page you could try: https://rawgit.com/pkra/ac7d48279728d3ab5881/raw/bb676922f161233224a1aa73b3ba9caf53539225/mathjax-assistivemml.html

sinabahram commented 9 years ago

On that page (great page, by the way, nice tests),

Jaws 16, latest rev, IE 11, latest patches, win 7 pro, results in them being read as well as Jaws can. Nothing on your end to fix there.

However, same setup, but with NVDA, latest nightly, NVDA first reads the raw mathml representation that would have been read such as “a question mark 0” before reading “a not equal to zero”

pkra commented 9 years ago

@sinabahram thanks!

Jaws 16, latest rev, IE 11, latest patches, win 7 pro, results in them being read as well as Jaws can. Nothing on your end to fix there.

Fair enough.

However, same setup, but with NVDA, latest nightly, NVDA first reads the raw mathml representation that would have been read such as “a question mark 0” before reading “a not equal to zero”

Ok. I'll check out IE11 some more.

pkra commented 9 years ago

One more data point.

dpvc commented 9 years ago

One question I have is about @jcsteh's concerns about accessing the MathJax menu. The current hidden MathML is inside a container that also has the HTM-CSS output (with aria-hidden="true"), and that container has the event handlers for the contextual menu (and a click handler for when the contextual menu is set to pass through to the browser contextual menu). Does this make it possible to access the MathJax contextual menu in screen readers? Does this cause other side effects? I have had VoiceOver read "clickable" after every term in the MathML, but I admit it is a very old version of VoiceOver, so things might work better now.

I'm not entirely sure what to do about users who are working mouse-less and need to use tab to access things like menus. What is the right way to give them access to the MathJax menu without causing problems for other users? (Perhaps I should start a new issue for this?)

pkra commented 9 years ago

(Perhaps I should start a new issue for this?)

There's #939 and #1093 but maybe a meta issue is suitable.

dpvc commented 9 years ago

Manual navigation (tapping on an equation, double tapping to go fullscreen) did not work.

Can you be more specific about what this is supposed to do? Is tapping on the expression supposed to read it? What does "go full screen" mean in terms of a screen reader? Do you want the whole math expression to be zoomed to full screen size? (This doesn't seem to involve the screen reader, so I'm a little confused.)

dpvc commented 9 years ago

(Perhaps I should start a new issue for this?)

There's #939 and #1093 but maybe a meta issue is suitable.

I was hoping for comments from the AT community (who have been active here), which is why I posted here first. I'm not sure any of them are following those issues.

pkra commented 9 years ago

Can you be more specific about what this is supposed to do?

Tapping on an element reads it out loud. For math you get a friendly reminder at the end of your reading to "double tap to zoom"; double tapping then makes the equation full screen, it goes on to read the first subexpression, and then you can iterate that with subexpressions (tap to read, double tap to zoom etc).

I'm not sure any of them are following those issues.

Right. Just wanted to point out that those issues exist.

dpvc commented 9 years ago

Thanks for the extra info about double-tapping. I wonder if it would be possible for MathJax to transfer the tap events to the MathML element via some sort of dispatchEvent() call?

sinabahram commented 9 years ago

A critical piece is not mentioned here, which is swiping to it. so take one finger and swipe rapidly from left/right or right/left. You can think of this as tab and shift+tab. I would touch close to the math and then begin swiping.

pkra commented 9 years ago

@sinabahram ah, I did not know that -- thanks.

So that sliding works, too. The math is read is out and the visual rendering is skipped.

You can additionally double-tap the highlighted area to trigger full screen exploration (though you obviously get Safari/WebKit rendering at that point).

sinabahram commented 9 years ago

Understood. that makes sense.

fsrg commented 9 years ago

Davide, I'm trying to get a more in-depth answer for you on this; so a response may not be immediate.

Thanks, RG

dpvc commented 9 years ago

@fsrg, Thanks! Your input will be greatly appreciated.

cannona commented 9 years ago

The math seems to read correctly in Firefox with Jaws 16.

Aaron

pkra commented 9 years ago

Thanks for letting us know, @cannona!

fsrg commented 9 years ago

Davide, I don’t think there is a great way to give programmatic access to the context menu that will work in all browsers. I wonder though if a hot key might work? As long as the key is unlikely to be used by the chrome or the page it’s embedded in, it might be an acceptable way for users to access the MathJax context menu.

Thanks, RG

dpvc commented 9 years ago

@fsrg, the contextual menu is equation-specific, so if the key event listener was bound to the container element just like the current menu listener, then that would probably work. That would also solve the issue for those using tabbing (if, we added a TABINDEX somewhere), right? Of course, that would mean that people would have to tab through all the math whenever they tab around the page. That probably is too much.

Would it be acceptable to have a hotkey that turns on/off the TABINDEX for the math elements (i.e., toggles their focusability)? Or is that also a privacy problem?

fsrg commented 9 years ago

Could you explain why turning on the tab stops with a key is a privacy issue?

dpvc commented 9 years ago

Could you explain why turning on the tab stops with a key is a privacy issue?

I wasn't claiming that it was, I was asking if it was. There was a lots of blow-back about suggesting that there be an opt-in for inserting the MathML into the page, and privacy was one reason (the page would know you are asking for assistive functionality). A setting that toggles TABINDEX might be considered by some to be the same type of thing. I'm not sure, which is why I asked. Sorry for the confusion.

sinabahram commented 9 years ago

As long as the tab lands on the math only once, then shouldn’t we just leave that as default?

Especially, given that the Math is interactive with most screen readers these days.

dpvc commented 9 years ago

As long as the tab lands on the math only once, then shouldn’t we just leave that as default?

Again, I'm asking for advice from the community. If a user is using tabbing to navigate among the various interactive pieces of the page, then would he or she really want to have to tab through all the math to get to that button near the bottom of the page? On the other hand, if they don't then how will they be able to select the math element for which the hotkey would be active? I don't do that kind of browser enough myself to make a good decision about that, and was hoping those with more experience would be able to give us some guidance.

fsrg commented 9 years ago

I’m not sure how tabbing will work, since the browser won’t be able to turn the MathJax control into an MSAA or UIA role/control type. But maybe if you put an ARIA role on it we can get the type information that way.

Thanks, RG

sinabahram commented 9 years ago

That’s exactly what I’m thinking. In fact, a math role would be appropriate here?

pkra commented 8 years ago

Just an update for people on this thread. We've finished work on #939 (making the MathJax menu accessible).

I've updated the example mentioned earlier. We'd appreciate any and all feedback!

pkra commented 8 years ago

Ack -- @dpvc I just realized that the AssistiveMML branch is not yet merged into develop. So the updated demo is missing the extension. When do you think that will be merged?

dpvc commented 8 years ago

I want to fix the aria stuff first, since the extension currently has to manipulate those (which was just intended as a temporary solution). I hope to be able to work on it this weekend.

pkra commented 8 years ago

@dpvc no problem. I've reverted my change to the AssitiveMML demo (it might take a while until rawgit picks up on the change -- sorry about that).

I've created a separate demo for the menu at https://rawgit.com/pkra/4c57afc70c4f1a4fb356/raw/450265ccd8ed8ccfc78d2ad63fe988f96c4bc915/mathjax-accessiblemenu.html

sinabahram commented 8 years ago

That sample page does not work with any combination I tried. The four I tried quickly were NVDA+Firefox, Jaws16+Firefox, NVDA+IE11, and Jaws16+IE11.

I wasn’t able to read any of the mathematics as it did not show up as mathml on the page.

fsrg commented 8 years ago

FYI: JAWS will only use MathJax in IE.

Thanks, RG

pkra commented 8 years ago

Thanks for the feedback!

That sample page does not work with any combination I tried.

This is totally my fault. Since I now mentioned two (three, four) different pages in the thread I can't tell which page you might refer to. My fault -- sorry about that!

Trying to recap:

Thanks for any additional feedback!

sinabahram commented 8 years ago

Ok, three out of four:

NVDA+Firefox works

Jaws+IE and Jaws+firefox work

However, NVDA+IE does not.

At a client site all week, but as I grab few moments here and there, I can test the menu and anything else you throw on here. thank you for all of your work on this!

sinabahram commented 8 years ago

No, Jaws works fine with firefox as of a few updates ago. I can provide links if that’s helpful.

pkra commented 8 years ago

Re @fsrg

FYI: JAWS will only use MathJax in IE.

Actually, we had good results with JAWS on Firefox using the new extension, cf. https://github.com/mathjax/MathJax-docs/issues/111

Re @sinabahram

However, NVDA+IE does not.

That's #1235 and seems to be caused by NVDA.

@sinabahram does the menu work for you (to some degree)?

fsrg commented 8 years ago

JAWS does support MathML in Firefox. But it does not access content via MathJax in Firefox.

Thanks, RG

pkra commented 8 years ago

JAWS does support MathML in Firefox. But it does not access content via MathJax in Firefox.

Well, it does work anyway with the AssistiveMML extension :-)

dpvc commented 8 years ago

FYI: JAWS will only use MathJax in IE.

This sounds like JAWS is detecting MathJax in IE and calling on it directly? In that case, will the added MathML in the DOM cause problems (like reading the math twice) for JAWS in IE?

fsrg commented 8 years ago

I don’t think so, provided that the MathML in the dom is inside the MathJax element JAWS looks for when determining if it has encountered MathJax. As I recall, it is one of the top level elements.

Thanks, RG

pkra commented 8 years ago

According to feedback from other JAWS users and my own testing, JAWS does not read things twice in IE. See the MathJax-docs issue for further details.

dpvc commented 8 years ago

OK, great! That sounds good (the MathML is inside the top-level MathJax element container).

dpvc commented 8 years ago

I've merged the assistive-mml branch into develop.