met4citizen / TalkingHead

Talking Head (3D): A JavaScript class for real-time lip-sync using Ready Player Me full-body 3D avatars.
MIT License
350 stars 108 forks source link

Laptop struggles, any optimizations I could try? #26

Closed JPhilipp closed 7 months ago

JPhilipp commented 7 months ago

The head runs fine on my main PC, but when using Twitch Studio (and having my server run in the background), my laptop struggles a bit and adds flicker in the stream when the head moves.

What are some things I might try to optimize performance?

Via CSS, I use an additional blur of 0.5 and slight background shadow on the avatar div, too. It probably doesn't help either (I'll try disable it now). Edit: That didn't really help.

Thanks!

met4citizen commented 7 months ago

Running a real-time full-body 3D animation takes a lot of resources, and there is a lot going on in your app already, so I'm not too surprised.

One thing that might help is to reduce the quality of the 3D model. When you download your model from Ready Player Me, you can try, for example, adding the parameters lod=1 (50% reduction in triangle count) and/or textureAtlas=512 (reduces the number of draw calls). For a full list of options, see:

https://docs.readyplayer.me/ready-player-me/api-reference/rest-api/avatars/get-3d-avatars

Note: You must always include all the morph targets, since they are needed for lip-sync and facial expressions.

All you can do with the TalkingHead class is to lower the modelFPS (default is 30). The quality of animation, of course, suffers. While testing, you can display the FPS statistics screen to see if there is a specific time when the frame rate fluctuates. Add these values to the options when creating the TalkingHead instance:

modelFPS: 25,
statsNode: document.body,
statsStyle: "position: fixed; bottom: 0px; left: 0px; cursor: pointer; opacity: 0.9; z-index: 10000;"
JPhilipp commented 7 months ago

Thank you! I fear it might be something more with Twitch Studio, as the animation in the browser window is very smooth. It's just the preview within Twitch Studio and the Twitch stream which then introduces lag and flicker, it seems.

I'll investigate some more with the FPS counter and possibly decreasing FPS, Cheers.

met4citizen commented 7 months ago

One more thing that is good to know and keep your eye on. If you are using subtitle callbacks or those timed markers, you should make the callback functions as short as possible. Those callbacks are called from within the animation loop, so you shouldn't do any heavy lifting there. There are times when you can't help it, and it is normal to see occasional log entries [Violation] 'requestAnimationFrame' handler took 58ms (e.g. when changing the avatar), but you shouldn't get them all the time.