search

TMElyralab / MuseTalk

MuseTalk: Real-Time High Quality Lip Synchorization with Latent Space Inpainting
Other
2.97k stars 369 forks source link

Lips and speech out-of-sync #140

Open xiankgx opened 4 months ago

xiankgx commented 4 months ago

Dear authors, wonderful work.

I tried to use the samples from video-retalking.

The generated video looks good, however there seems to be out-of-sync issue.

For example, see this:

Input video: https://github.com/OpenTalker/video-retalking/blob/main/examples/face/1.mp4 Input audio: https://github.com/OpenTalker/video-retalking/blob/main/examples/audio/1.wav

Output https://github.com/TMElyralab/MuseTalk/assets/4113258/bc2b9cef-6fa4-41c3-b639-4495230853db

ZRSN-1101 commented 2 weeks ago

I have the same problem. Have you found a way?

eoffermann commented 2 weeks ago

You can fix it as a post-process using https://github.com/joonson/syncnet_python

The MuseTalk repo and publicly released model were not trained with robust lipsync during training. The paper indicates that TME Lyra Lab used an additional loss function during training that leveraged SyncNet. In fact, the lip sync loss function was weighted 3x as much as the other loss functions in their training, but that's not reflected in the model available from this repo.

KISHAN8888 commented 1 week ago

nothing improves even after doing this step!!! any alternatives?