Add an option to read the text of an exploration to the learner

[seanlip]

From #159:

Create a setting within an exploration to allow the text of the content to be read within the browser. This would be useful for creating math/science explorations that do not require the learner to be a high-level reader.

[seanlip]

/cc @coryroush (original reporter), @jg1141 (thread participant)

[seanlip]

@edallison: this is a little speculative, but I think it would be a fun project to try (that's also somewhat larger than the other ones). I think this is something that would be great for younger learners, or learners for whom English isn't their first language.

[coryroush]

Text-to-speech functions are pretty common for applications and utilities similar to this that could be used for independent learning in a classroom.

[seanlip]

@edallison: fyi, this might be useful! https://developers.google.com/web/updates/2014/01/Web-apps-that-talk-Introduction-to-the-Speech-Synthesis-API?hl=en

[edallison]

Thanks Sean. I'm glad you brought this up as it is actually the API I've been looking at implementing(at least initially). I was however a little concerned with cross browser compatibility. Would you think this would be a better solution as opposed to a separate third party library?

[seanlip]

I think it depends on what's available -- last time I checked, support for speech-related APIs was all over the map. I'd say the ideal would be a solution that works in all browsers, but if we can't get to that, then a solution that just works in Chrome is better than no solution. What alternatives have you found?

[edallison]

A couple I have looked at are http://www.cstr.ed.ac.uk/projects/festival/ and http://responsivevoice.org/ though I still haven't done enough research to see how viable they might be. In the meantime I will work on a PR for the Web Speech API you mentioned as it seems to be relatively straightforward. Once that is working I'll continue exploring some more versatile solutions.

[seanlip]

It looks like Festival won't work -- it's written in C++, which App Engine doesn't support. However, responsivevoice looks promising.

I agree with building out the Web Speech API version as a first step, though. It should be straightforward to replace it with other third-party libs later.

[jg1141]

Here I've got the HTML5 SpeechSynthesisUtterance going for single words in JavaScript:

function playWord(word) { var msg = new SpeechSynthesisUtterance(word); window.speechSynthesis.speak(msg); };

For example, click P to put C16 on the phoneme bar, then the magnifying glass (search) to find the words starting with phoneme C16, then one of them, such as PLATFORM.

Voice used is the one available on my local PC (can be changed through settings on a Mac).

http://phoneme-dictionary2.appspot.com/js/english_sound_map.html

[jg1141]

I found three likely places to insert text-to-speech JavaScript code, but have no idea where to put the toggle that would turn this feature on and off.

In /oppia/core/templates/dev/head/player/PlayerServices.js

      var questionHtml = makeQuestion(initialState, [startingParams]);
      // Speak initial state content
      var speak_text = new SpeechSynthesisUtterance(initialState["content"][0]["value"].replace(/<\/?[^>]+(>|$)/g, ""));
      window.utterances = [];
      utterances.push( speak_text );
      speechSynthesis.speak(speak_text);
      if (questionHtml === null) {
        alertsService.addWarning('Expression parsing error.');
        return;
      }

and

          var feedbackHtml = makeFeedback(outcome.feedback, [oldParams]);
          // Speak outcome content
          var speak_text = new SpeechSynthesisUtterance(outcome.feedback[0].replace(/<\/?[^>]+(>|$)/g, ""));
          window.utterances = [];
          utterances.push( speak_text );
          speechSynthesis.speak(speak_text);

Finally, for the terminal card, in /oppia/core/templates/dev/head/player/ConversationSkinDirective.js

        $scope.showPendingCard = function(
            newStateName, newParams, newContentHtml) {
          if (newContentHtml.startsWith("Congratulations, you have finished")) {
              // Speak terminal state content
              var speak_text = new SpeechSynthesisUtterance("Congratulations, you have finished!");
              window.utterances = [];
              utterances.push( speak_text );
              speechSynthesis.speak(speak_text);
          };

[seanlip]

Ah -- yep, we've got the technical groundwork mostly sorted for this one, I think, but the part we're having trouble with is figuring out the right way to display it to the learner. You might find the discussion in #1115 interesting!

[kevinlee12]

Hi @edallison, any updates for this? thanks!

[seanlip]

Ah, actually, this is still WIP in #1115. I think we need input from someone else with regards to the design aspects.

[kevinlee12]

Ah, thanks!

On Mon, Jul 25, 2016 at 1:36 PM, Sean Lip notifications@github.com wrote:

Ah, actually, this is still WIP in #1115 https://github.com/oppia/oppia/pull/1115. I think we need input from someone else with regards to the design aspects.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/oppia/oppia/issues/902#issuecomment-235076633, or mute the thread https://github.com/notifications/unsubscribe-auth/AFfdMKyo9Bk3hs4c8P7fV_R_YFhpihNnks5qZR5IgaJpZM4FsUmb .

[seanlip]

Note to contributors: please see prior work in #1115.

[seanlip]

Closing. This is something @tjiang11 and I have worked on recently, though in somewhat different form and for multiple languages.