search

hecomi / uLipSync

MFCC-based LipSync plug-in for Unity using Job System and Burst Compiler
https://tips.hecomi.com/entry/2022/01/30/152519
MIT License
1k stars 123 forks source link

Using uLipSync in Editor mode #2

Open lehtiniemi opened 3 years ago

lehtiniemi commented 3 years ago

I was wondering if there could be support for using and previewing lip sync in Editor mode as well?

I tried the following setup, unable to make the mouth move: -create timeline -add audio track -setup uLipSync on a scene game object -drag the audio source from the object to the timeline Audio track slot -drag an audio clip onto the timeline -hit play

The audio plays, but uLipSync doesn't trigger.

Am I doing something wrong or is Editor mode currently not supported? In this case, seeing this feature would be really useful!

lehtiniemi commented 3 years ago

I don't know what I was doing, but it seems to be working after adding [ExecuteAlways]-tag to uLipSync class.

lehtiniemi commented 3 years ago

@hecomi I know this is more like a feature request, but got back to this now and suddenly I can't make the [ExecuteAlways] to work. I'm now trying without Timeline, by adding the clip directly to an AudioSource that is in the same object.

Do you know what could be going on? I'm not getting callbacks from uLipSync, only when I hit play and when I hit stop. I checked and OnAudioFilterRead gets constantly triggered, but something with the analysis doesn't happen because the callback doesn't deliver the results.

It would be so cool to have your script to work in Editor mode as well - a lot of animation testing happens in Editor mode and it would be really handy to see the lipsync included with the rest of the animation.

lehtiniemi commented 3 years ago

I don't know if this is relevant, but AllocateBuffers that is called at OnEnable seems to assume that profile is set. However, when I set ExecuteAlways, OnEnable is called immediately upon adding the component, but at this point profile-field is empty. Is something allocated incorrectly in this case? For the editor mode to work, should buffer allocation be made each time the profile is changed?