NVDA not properly present an image when role="math" is present

GeorgeKerscher commented 9 months ago

Description:

In digital books, many authors and publishers include math by using an image. While MathML is recommended, some will still deliver an image with alt text. When the role on the image is set to math, i.e., role="math" NVD is notperforming as expected. We would expect:

You should be able to do first letter navigation, i.e., press "g" to go to the graphic.
You should be able to read the alt text by word or character.

Steps to reproduce:

visit Test for ARIA Math Role on an Image
Read the page, which contains an image of a math expression that includes alt text. It has a role="math".
Notice that the alt text is read as a single line.
Try to read the alt text by word or character, which you cannot do.
Go to the top of the page and in browse mode using single letter navigation hit "g" to go to graphic, and you will not find the image.

Actual behavior:

Example speech output from the sample referenced.

This marks the spot where speech viewer started to capture the output. The page is visited and we get: Begin speech viewer output enter Test for ARIA Math Role on an Image document heading level 1 Test for ARIA Math Role on an Image up arrow heading level 1 Test for ARIA Math Role on an Image down arrow heading level 2 Introduction down arrow This test is to evaluate how assistive technologies interpret an image with math role and down arrow alternative text. down arrow heading level 2 Test down arrow Quadratic formula: x=⟮−b±√⟮b²−4ac⟯⟯÷2a
right arrow space ctrl+right arrow space ctrl+home heading level 1 Test for ARIA Math Role on an Image no next graphic

End speech viewer output

Notice that the alt text is read, but cannot be navigated. Also notice that with single letter navigation, the graphic (img) is not found.

Expected behavior:

The expected behavior is to have the ability to read alt text by character and word. Also, because it is a graphic, you should be able to go to it by hitting the letter G.

NVDA logs, crash dumps and other attachments:

Not aplickable.

System configuration

NVDA installed/portable/running from source:

NVDA is installed on this computer.

NVDA version:

Windows version:

About Windows Microsoft Windows Version 23H2 (OS Build 22631.2861) ©

Name and version of other software in use when reproducing the issue:

This behavior is with the browsers Chrome, Edge, and Firefox latest version. The Chrome version is: Version 120.0.6099.199 (Official Build) (64-bit)

Other information about your system:

Other questions

Does the issue still occur after restarting your computer?

Yes

Have you tried any other versions of NVDA? If so, please report their behaviors.

We noticed this vehavior on the previous version of NVDA as well.

If NVDA add-ons are disabled, is your problem still occurring?

Yes, when all add-ons are disabled the behavior is the same.

Does the issue still occur after you run the COM Registration Fixing Tool in NVDA's tools menu?

Yes!

Adriani90 commented 9 months ago

This michgt be a duplicate of #13702. or #5862. Both indicate that this needs an external fix.

lukaszgo1 commented 9 months ago

Setting role of an object to math means that the original role of graphic cannot be conveyed to ATS - HTML has no concept of multiple roles. Why these images are given a role of math in the first place? If this is only to have them announced as math, you can provide alternative role description with aria-roledescription, but in general this is not recommended. When I'd hear that something is math then this implies that I can interact with this element using MathPlayer and have the content rendered in Braille, so it is better to report that the element is a graphic, and present the content in the alt text without changing the role.

GeorgeKerscher commented 9 months ago

this issue is not a duplicate of those two. They relate to the presence of MathML. This is addressing the content of an image. It is somewhat related, because if publishers mark the images of math with the role="math" then we would have more opportunities to convert the image of math into proper MathML through a post process.

GeorgeKerscher commented 9 months ago

I agree that the best practice for authors and publishers is to use MathML, but there are many cases where this guidance is not being followed. We have written a position statement on this. [Position Paper on using MathML] (https://daisy.github.io/transitiontoepub/information-sharing/position-paper-plain-text-math/)

It seems that placing role=math" on an image of math conveys the semantics. If this attribute is set then organizations that provide materials to persons with disabilities could easily identify the images of math and convert it to MathML. However, if we make a best practice recommendation of the fallback for publishers that produce images of math to use this attribute, then NVDA should treate it as an image and allow navigation to it and review of the alt text.

lukaszgo1 commented 9 months ago

It looks like this is a supported use case according to this W3C documentation. I'm not sure if I personally agree with this, as it seems to be abusing the role concept, but that is probably a discussion for a different place. Note that even when we will make the alt text reviewable making sure that these can be navigated to with the g key is going to be tricky, as we don't support multiple roles for a single object. There is also a question of how to report such objects, as math or as an image. Have you investigated how other screen readers behave here? cc @jcsteh

GeorgeKerscher commented 9 months ago

I just checkected with Jaws 2024.3 and you can get to the image with the g single letter navigation. The alt text can also be reviewed with the arrow keys.

I will wait for others on our team to respond about other platforms.

lukaszgo1 commented 9 months ago

What role is reported for this object by JAWS - math, or image?

clapierre commented 9 months ago

BTW on a Mac with Voice Over when you have an image with alt text describing the math and role="math" the results are mixed.

Safari with Voice Over you hear the alt text description followed by "empty math" eg: "x+5=7 empty math" where the image alt="x+5=7" role="math" was provided.

Chrome with Voice Over all you hear is "math" and not the alt text description.

gregoriopellegrino commented 7 months ago

To add context. On Android using Chrome + Talkback the screen reader reads the alt-text content followed by "math element" (using a different voice than the one used for the main content).

For example "x+5=7 math element".

nvaccess / nvda