Hypertext-to-Speech

AdamSobieski commented 6 years ago

I would like to share some ideas with respect to advancing the Web Speech API. The ideas pertain to hypertext-to-speech, the speech synthesis of XHTML.

Hypertext-to-speech involves processing XHTML DOM trees with subtopics including SSML attributes, CSS Speech Module properties and new JavaScript events such as onsynthesis.

<html xmlns:ssml="http://www.w3.org/2001/10/synthesis">
  <head>
    <script type="text/javascript" src="..."></script>
    <style>
      [role="topic"] { voice-stress: strong; }
    </style>
  </head>
  <body ssml:alphabet="ipa">
    <p>
      <span id="sentence-1" onsynthesis="function(event);"><span ssml:ph="/ðɪs/" role="topic">This</span> is a sentence of <span ssml:ph="/tɛkst/">text</span> with markup for hypertext-to-speech.</span>
    </p>
  </body>
</html>

JavaScript API

speechSynthesis.speak(document);

speechSynthesis.speak(document.getElementById('sentence-1'));

var fragment = document.createDocumentFragment();
...
speechSynthesis.speak(fragment);

var doc = document.implementation.createDocument('http://www.w3.org/1999/xhtml', 'html', null);
...
speechSynthesis.speak(doc);

References

foolip commented 6 years ago

@AdamSobieski, is speechSynthesis.speak(document) something that could be implemented on top of the current API, are are there things that are impossible to do with the current API that would be needed first?

It sounds a lot like you're describing a screen reader, how would an API to read a whole document interact with that?

AdamSobieski commented 6 years ago

@foolip , I think that such a polyfill is possible. Algorithms can traverse DOM trees to produce SSML. In theory, a hypertext-to-speech JavaScript library could be implemented atop the current Web Speech API.

There are some complexities. There is, for instance, complexity with respect to the speech synthesis of MathML. One solution is to use the MathML annotation framework. That is, inside of <annotation-xml> elements, one can put SSML versions of MathML expressions.

<html xmlns:ssml="http://www.w3.org/2001/10/synthesis">
  <body>
    <span id="sentence-2">This is a sentence with mathematics <math>...<annotation-xml encoding="application/ssml+xml">...</annotation-xml>...</math> for hypertext-to-speech.</span>
  </body>
</html>

Screen readers and accessibility are proximate topics. A hypertext-to-speech API could clarify a number of interoperating technologies which are also utilized by screen readers. A hypertext-to-speech API (client-side speech synthesis) and media overlay standards (server-side speech synthesis) could simplify Web-based multimodal dialogue systems.

References

foolip commented 6 years ago

What I'd suggest here is to experiment with achieving the effect that you're after using JavaScript. If there are things not possible with existing web platform APIs, file requests for those smaller missing bits. (Note that in theory Web Speech API already supports SSML as a string to the SpeechSynthesisUtterance constructor, although it doesn't work in Chrome and I'm not sure about other browsers.)

Since not everybody in the Speech API Community Group subscribes to GitHub issues, you could also email public-speech-api@w3.org to reach more people who have been involved with Web Speech. If you are interested in pursuing a high-level feature like this, however, I would ask that it's first incubated in https://wicg.io/, as it would be a significant expansion of scope for this spec.

foolip commented 6 years ago

I added a test for SSML in https://github.com/web-platform-tests/wpt/pull/12568 to see if it's supported anywhere.

WICG / speech-api