play emojis in text

[branaway]

I've got some emojis embedded in the text to be spoken. I have used Azure TTS to get all the audio offsets for them. And I have mixed the emojis in the word list of the speach data. How do I schedule emojis animations? It's not obvious. I don't think I can use the speakEmoji() method directly, can't I?

Thanks!

[branaway]

I must play the emojis only when the audio playback reaches the right time points.

[met4citizen]

One way is to split your text/audio into two, and then call speakAudio, speakEmoji, speakAudio. However, based on what you told me, your best option is to use timed markers, so I will focus on that here.

You can add timed markers directly to the audio object that you pass to speakAudio by using arrays markers and mtimes. In these timed callback functions, you can start the emoji animation. You can't, however, as you said, just call the speakEmoji method, because instead of starting the animation right away, it only adds a new emoji item to the speech queue, and you don't want that. You need to add the animation directly to the animation queue.

Here is an example that hopefully gives you the general idea:

// Play emoji
function playEmoji(emoji) {

  // Find the right animation template
  let animTemplate = head.animEmojis[emoji];
  if ( animTemplate && animTemplate.link ) {
    animTemplate = head.animEmojis[animTemplate.link];
  }

  if ( animTemplate ) {

    // Look at the camera for 500 ms (optional)
    head.lookAtCamera(500);

    // Add the animation to the animation queue
    const anim = head.animFactory( animTemplate );
    head.animQueue.push( anim );
  }
}

...

// Add a timed emoji animation to the audio object
audio.markers = [ playEmoji.bind(null,'😐') ];
audio.mtimes = [ 100 ];

Note that these timed markers get called from inside the animation loop, so you should avoid doing any heavy processing inside the callback function.

[branaway]

I put the emojis in the words stream of the current speech data, then modified the subtitle callback to playEmoji(). It seemed to work. The emojis were played back immediately. How is this hack?

[met4citizen]

Yes, that is certainly one way to do it.