Poor tracking when using EchoMimic as Driving Video

[Sidd065]

I am using EchoMimic's video output as the driving video for LivePortrait.

This is the Driving Video:

https://github.com/user-attachments/assets/f5d41ee9-51a0-4f74-a539-51ff9a5dd816

This is the LivePortrait output:

https://github.com/user-attachments/assets/34a79f98-b934-4ec1-bc9e-09a34eb1a291

https://github.com/user-attachments/assets/2e559f66-3ae5-4689-9436-3fe279b11f01

When using any normal driving video, the output is fine, this only happens with EchoMimic videos.

I suspect its the artefacts and noise in EchoMimic's output that's causing this. Is there any pre-processing I can do to get better results from LivePortrait?

[cleardusk]

See https://github.com/KwaiVGI/LivePortrait?tab=readme-ov-file#driving-video-auto-cropping, a guide about driving video. @Sidd065

[NikoBele1]

i had similar problems, when your driving video is not neutral, Live Portrait still treats it as neutral and when your face in the driving video closes the mouth, your face in the source frame will close it aswell, even if it is closed already, thats why you see it so weirdly exaggerated. Same with eyes, if you have closed eyes in the driving video, it will exaggerate opening them in the source frame animation.

[O-O1024]

Make sure the first frame of driving video is a frontal face with neutral expression, @Sidd065 The first frame of your driving video is with open mouth expression. Try and show the result! @Sidd065

[galigaligo]

我也在做这个工作，在EchoMimic 的驱动音频前加200ms空白，就会有好转