met4citizen / TalkingHead

Talking Head (3D): A JavaScript class for real-time lip-sync using Ready Player Me full-body 3D avatars.
MIT License
349 stars 107 forks source link

Proper way to unload old avatar before loading new? #25

Closed JPhilipp closed 7 months ago

JPhilipp commented 7 months ago

What's the proper way to unload the old avatar, when one wants to dynamically change to a new one during live usage? I'm currently simply nulling things -- see below -- but e.g. I want the stacked speech queue to also immediately stop etc. Thanks!

let head;
let audio;
let avatarUrl;

export async function avatarLoad(id, language, gender, mood = 'neutral') {
  const url = `/avatars/${id}.glb`;
  if (url != avatarUrl) {
    console.log('Loading avatar', id, language, gender);
    avatarUrl = url;

    const nodeAvatar = document.getElementById('avatar');

    // "Unloading" old:
    head = null;
    nodeAvatar.innerHTML = '';
    audio = null;

    head = new TalkingHead( nodeAvatar, {
      ttsEndpoint: "none",
      cameraView: "head",
      cameraRotateEnable: false,
      cameraRotateY: 0.4,
      lightAmbientColor: '#fff1d8'
    });

    try {
      await head.showAvatar( {
        url: url,
        body: gender,
        avatarMood: mood,
        lipsyncLang: language
      }, (ev) => {});
    } catch (error) {
      console.log(error);
    }

    console.log('Loading avatar done.');
  }
}
met4citizen commented 7 months ago

There is no need to create a new TalkingHead class instance if you just want to change the avatar. In fact, I would advise against it. You can just call the head.showAvatar again. If before that you want to stop speaking, you can call head.stopSpeaking(). It also clears the speech queue (unlike pauseSpeaking).

JPhilipp commented 7 months ago

Thanks, makes sense!

Trying out the new approach, there's one issue: The new avatar will be falsely positioned. In the old, object-instance-recreating version, the new avatar was better centered. Now e.g. blond-man Gunther is too high when he replace a less tall woman. Help please?

sample

export async function avatarLoad(id, language, gender, mood = 'neutral') {
  const url = `/avatars/${id}.glb`;
  if (url != avatarUrl) {
    console.log('Loading avatar', id, language, gender);
    avatarUrl = url;

    const nodeAvatar = document.getElementById('avatar');

    if (!head) {
      head = new TalkingHead( nodeAvatar, {
        ttsEndpoint: "none",
        cameraView: "head",
        cameraRotateEnable: false,
        cameraRotateY: 0.4,
        lightAmbientColor: '#fff1d8'
      });
    }
    else {
      await head.stopSpeaking();
    }

    try {
      await head.showAvatar( {
        url: url,
        body: gender,
        avatarMood: mood,
        lipsyncLang: language
      }, (ev) => {});
    } catch (error) {
      console.log(error);
    }
  }
}
met4citizen commented 7 months ago

I couldn't reproduce the issue in my test app. Make sure you are using the v1.1 or the latest. Also, check that the imported three.js version is v0.161.

The male figure is taller, but once the new avatar has been loaded, the class should automatically adjust the camera based on the new eye level:

https://github.com/met4citizen/TalkingHead/assets/76654952/275d7ea8-d796-49e8-bf6b-89e6927ddd04

met4citizen commented 7 months ago

I think I solved it. It seems that in my test app, there is, in fact, an explicit call to setView. I didn't make the adjustment automatic because I wanted to leave the decision to the app. So, after the call to showAvatar, just reset the camera view by calling head.setView('head').

JPhilipp commented 7 months ago

head.setView('head') works great, thanks!

Not so important, but any way to make the new camera view instant, and not the quick smooth transition?

met4citizen commented 7 months ago

Currently, that is not configurable, but you can try the following. This is a hack, so I haven't tested it, and it might not work in the upcoming versions.

head.setView('head');
head.cameraClock = 999; // Hack
JPhilipp commented 7 months ago

Cheers!