Direct line speech channel and selectVoice webchat option

BeeMaia commented 4 years ago

Version

latest version on CDN or npm 3.7.1 inside reactjs

Describe the bug

I have a problem inside a reactjs application that use direct line speech channel but I can reproduce running the sample https://github.com/microsoft/BotFramework-WebChat/tree/master/samples/03.speech/a.direct-line-speech, the user is not able to change or alter the voice pattern used after updating selectVoice parameters.

To Reproduce

Steps to reproduce the behavior:

Copy code from 03.speech/a.direct-line-speech sample into Web Chat enabled page
Apply this change:
Run bot and navigate to page
Observe text to speech voice pattern
Change language const value
Observe no change in voice pattern

Expected behavior

When changing language parameters, the voice pattern used in web chat for text to speech should change

[Bug]

compulim commented 4 years ago

I think you are talking about 4.7.1.

The repro steps is detailed and looks legit. 👍🏻

Will triage and hope this fix in next release (R10).

compulim commented 4 years ago

Hi @BeeMaia.

Direct Line Speech do the speech synthesis on the bot part, before reaching Web Chat. Thus, selectVoice will not work in this case. I will update our docs to indicate this.

However, you can change the voice on the bot side. I did a quick experiment and you can see the code below.

In short, it send SSML tag with a specific voice (en-US-JessaNeural and ja-JP-Ayumi-Apollo). And DLSpeech will synthesis the bot's message using that voice before reaching Web Chat.

As you can see in the code, you can synthesis a single message with two different voices even they are with different languages.

For details of the SSML, you can find it here, https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/speech-synthesis-markup?tabs=csharp.

class EchoBot extends ActivityHandler {
  constructor() {
    super();
    // See https://aka.ms/about-bot-activity-message to learn more about the message and other activity types.
    this.onMessage(async (context, next) => {
      const replyText = `Echo: ${context.activity.text}`;
      const speakText = `<speak
  version="1.0"
  xmlns="https://www.w3.org/2001/10/synthesis"
  xmlns:mstts="https://www.w3.org/2001/mstts"
  xml:lang="en-US"
>
  <voice name="en-US-JessaNeural">
    <mstts:express-as type="cheerful">That'd be just amazing!</mstts:express-as>
  </voice>
  <voice name="ja-JP-Ayumi-Apollo">
    <prosody pitch="+150%">素晴らしい!</prosody>
  </voice>
</speak>`;

      await context.sendActivity(MessageFactory.text(replyText, speakText));
      // By calling next() you ensure that the next BotHandler is run.
      await next();
    });

    this.onMembersAdded(async (context, next) => {
      const membersAdded = context.activity.membersAdded;
      const welcomeText = "Hello and welcome!";
      for (let cnt = 0; cnt < membersAdded.length; ++cnt) {
        if (membersAdded[cnt].id !== context.activity.recipient.id) {
          await context.sendActivity(
            MessageFactory.text(welcomeText, welcomeText)
          );
        }
      }
      // By calling next() you ensure that the next BotHandler is run.
      await next();
    });
  }
}

I will leave this bug open until we fix our docs.

compulim commented 4 years ago

To-do

[ ] Fix our doc to indicate selectVoice will not work with Direct Line Speech, instead, devs will need to use SSML tag instead
- References of SSML at https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/speech-synthesis-markup

microsoft / BotFramework-WebChat