search

edrlab / thorium-reader

A cross platform desktop reading app, based on the Readium Desktop toolkit
https://www.edrlab.org/software/thorium-reader/
BSD 3-Clause "New" or "Revised" License
1.86k stars 157 forks source link

TTS not pronouncing minus sign (−) in <text> elements of inline SVG and improperly grouping separate numbers together #2629

Closed mlouie-prcvi closed 3 weeks ago

mlouie-prcvi commented 3 weeks ago

I'm working on producing an EPUB for a math textbook, and am testing whether I can represent number lines and coordinate grids as inline SVGs in an EPUB file. Hopefully this would allow students to navigate the separate components of these diagrams.

While doing this testing, I noticed some issues with Thorium's inbuilt TTS. For example:

(1) The minus sign (−) is ignored so that the negative integers on the number line are not read as negative numbers, and (2) The labels -1 and 0 (represented as separate tags in the SVG) are improperly combined by the TTS and read as "ten"

Here is an example of the SVG code I have been using for a basic number line that shows these issues:

<svg width="100%" height="100%" viewBox="0 0 900 100" xmlns="http://www.w3.org/2000/svg">
    <!-- Draw the number line -->
    <line x1="50" y1="50" x2="850" y2="50" stroke="black" stroke-width="2"/>
    <text x="50.0" y="70">−8</text>
    <line x1="50.0" y1="45" x2="50.0" y2="55" stroke="black"/>
    <text x="100.0" y="70">−7</text>
    <line x1="100.0" y1="45" x2="100.0" y2="55" stroke="black"/>
    <text x="150.0" y="70">−6</text>
    <line x1="150.0" y1="45" x2="150.0" y2="55" stroke="black"/>
    <text x="200.0" y="70">−5</text>
    <line x1="200.0" y1="45" x2="200.0" y2="55" stroke="black"/>
    <text x="250.0" y="70">−4</text>
    <line x1="250.0" y1="45" x2="250.0" y2="55" stroke="black"/>
    <text x="300.0" y="70">−3</text>
    <line x1="300.0" y1="45" x2="300.0" y2="55" stroke="black"/>
    <text x="350.0" y="70">−2</text>
    <line x1="350.0" y1="45" x2="350.0" y2="55" stroke="black"/>
    <text x="400.0" y="70">−1</text>
    <line x1="400.0" y1="45" x2="400.0" y2="55" stroke="black"/>
    <text x="450.0" y="70">0</text>
    <line x1="450.0" y1="45" x2="450.0" y2="55" stroke="black"/>
    <text x="500.0" y="70">+1</text>
    <line x1="500.0" y1="45" x2="500.0" y2="55" stroke="black"/>
    <text x="550.0" y="70">+2</text>
    <line x1="550.0" y1="45" x2="550.0" y2="55" stroke="black"/>
    <text x="600.0" y="70">+3</text>
    <line x1="600.0" y1="45" x2="600.0" y2="55" stroke="black"/>
    <text x="650.0" y="70">+4</text>
    <line x1="650.0" y1="45" x2="650.0" y2="55" stroke="black"/>
    <text x="700.0" y="70">+5</text>
    <line x1="700.0" y1="45" x2="700.0" y2="55" stroke="black"/>
    <text x="750.0" y="70">+6</text>
    <line x1="750.0" y1="45" x2="750.0" y2="55" stroke="black"/>
    <text x="800.0" y="70">+7</text>
    <line x1="800.0" y1="45" x2="800.0" y2="55" stroke="black"/>
    <text x="850.0" y="70">+8</text>
    <line x1="850.0" y1="45" x2="850.0" y2="55" stroke="black"/>
    <circle cx="50.0" cy="50" r="4" fill="red"/>
    <circle cx="150.0" cy="50" r="4" fill="red"/>
    <circle cx="400.0" cy="50" r="4" fill="red"/>
    <circle cx="450.0" cy="50" r="4" fill="red"/>
    <circle cx="600.0" cy="50" r="4" fill="red"/>
    <circle cx="700.0" cy="50" r="4" fill="red"/>
    <circle cx="850.0" cy="50" r="4" fill="red"/>
</svg>

I have also tried embedding MathML as a foreign object in the SVG, but in this case:

(3) The TTS (with MathJax on or off) completely ignores the elements in the inline SVG and skips to the next section.

As a note, navigating the inline SVGs with Thorium and NVDA works extremely well. Thorium's inbuilt TTS also correctly reads the minus sign, or the dash sign as "negative" when I tried to reproduce the number line using just html and CSS. (I would prefer to use SVG over CSS, however.)

Here is a sample EPUB: InlineSVG-NumberLines-Sample.zip which has (i) An inline SVG number line illustrating issues (1) and (2) (ii) An inline SVG coordinate grid similarly illustrating issue (1) (iii) An inline SVG number line with embedded MathML, illustrating issue (3) (iv) An example of a number line created with just html/CSS where issue (1) and (2) don't arise

Is there a way that I could write the labels in the SVG code so that they are read aloud properly by the inbuilt TTS?

I'm using Thorium v3.0.0 on a Windows PC. (Although I noticed a similar pattern when using Thorium v3.0.0 on a Mac (i.e., (2) and (3) are both issues, and (1) manifests slightly differently in that minus is sometimes pronounced "negative", but sometimes pronounced "dash"). )

danielweck commented 3 weeks ago

Hello, Thorium supports the aria-label attribute on the svg element. Alternatively, Thorium also looks for a title child element that contains descriptive text. Failing this, the fallback is to parse the SVG markup and extract text from the descendants.

For MathML, Thorium looks for the alttext attribute and the fallback is the text content inside the Math markup. When MathJax is active (either by Thorium itself or because the Javascript was authored at source) Thorium looks for the aria-label attribute which is either authored statically or injected dynamically by MathJax's own text-to-speech "assistive" extension (failing that, there is a fallback to the aforementioned raw MathML alttext attribute).

danielweck commented 3 weeks ago

PS: I assume you are a representative of https://prcvi.org ? Thank you for your tests and feedback, this is always very helpful!

mlouie-prcvi commented 3 weeks ago

Yes, I am from the PRCVI; thanks for all you do too! Our current practice is actually to just use png/jpg images of the number lines and coordinate grids with alt-text, but I was hoping that Thorium's inbuilt TTS would be able to parse the SVG mark-up, so that students could see the labels on the axes and coordinates being highlighted while they are read out. This is the case when using Thorium with NVDA (which lets you navigate around the coordinate grid and number line labels), but I was worried students that don't use a screen reader would get confused when the negative symbol is ignored by the inbuilt TTS.

Is it impossible to support the aria-label attribute on the elements within the svg element?

danielweck commented 3 weeks ago

Hello, I fixed the SVG TTS readloud algorithm to correctly include deep-nested text fragments and foreignObject MathML. You will notice that this potentially causes an SVG island to be broken down into a great number of individual utterances, as demonstrated by your example (thank you for providing it!). This means that the user experience with the previous/next commands can be very repetitive and frankly annoying, and it forces usage of the mouse cursor in order to jump pass the currently-spoken SVG. The solution is "escapability", which typically applies to markup structures like table, table row, list, aside, endnotes, etc. The SHIFT+ALT keyboard modifiers can be used when clicking or activating the previous/next icon buttons in the audio playback toolbar at the top (there are keyboard shortcuts too for these commands). In your examples you will see this working well with the first two methods, but the third method (MathML) isn't escaping because in this case math markup is itself nested inside SVG. The fourth method (HTML) is a single utterance broken down into individual word boundaries so the escapability logic doesn't apply here.

mlouie-prcvi commented 2 weeks ago

Thanks so much for your amazingly quick response!

I tested out the new versions of Thorium (linux-intel, macos-intel, windows-intel), and my test epub was read out beautifully on the mac and linux versions! I noticed that there were some issues with the Windows version, however: it appears that while dashes (eg., -8) are read as "minus", the dedicated minus sign − is still being ignored, both when the dedicated minus sign is used in normal text, and when it's necessarily used within MathML.

Enthused by the new possibilities associated with being able to use inline SVGs with embedded MathML, I also noticed another issue with all three of the new Thoriums (linux-intel, macos-intel, windows-intel). With MathML (whether embedded in an SVG or not), the presence of parentheses seem to cause issues.

- Previous Thorium: the parentheses are properly read out, as are minus signs. - New Thorium: the parentheses are not read out, minus signs are also not read out (and on linux, in some cases, the plus signs are also not being read out...)

I've attached a modified EPUB that showed these issues; the first section has unembedded MathML with parentheses, and the second section has MathML with parentheses that has been embedded in an inline SVG. InlineSVG-Parentheses-2024-11-04.zip

Again, thanks so much for your quick response, it's really inspiring to know that we might be able to provide diagrams that are accessible in a way beyond adding alt-text!