search

Zejun-Yang / AniPortrait

AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animation
Apache License 2.0
4.64k stars 578 forks source link

strange face movements in mesh sequences from audio2mesh model #144

Open P2Oileen opened 6 months ago

P2Oileen commented 6 months ago

Nice Work! Thank you for opensourcing! 🙂 I have noticed that the face movements is quite strange in the mesh sequence output by audio2mesh model. For your information, you can see this video to see the mesh sequence: https://github.com/Zejun-Yang/AniPortrait/assets/26021786/b4b41380-217b-4e7c-ab81-c59e3bcd8bb7 The eyebrows area and the contours of the face have a strange distortion. I suspect this is a problem introduced because real human mediapipe facial landmark data collecting. However, in the final rendered video, although the landmarks in the middle are still distorted, the distortion becomes less noticeable in the final generated video. https://github.com/Zejun-Yang/AniPortrait/assets/26021786/594ed486-cbf1-4fe1-9632-579fe6df1bc5 I wonder if you have done any post-processing on the mesh sequence to make this distortion disappear, or is there a special smoothing design in your rendering process? Thank you for your reply!

P2Oileen commented 6 months ago

To add on, I want to highlight that the eye sockets will suddenly enlarge and shrink, creating a bizarre effect.

gessyoo commented 6 months ago

To add on, I want to highlight that the eye sockets will suddenly enlarge and shrink, creating a bizarre effect.

I've seen that exact issue too, and I'm wondering how to solve it. I've thought of fine tuning the model, but terabytes of training data is a bit intimidating, with only a single GPU to train with.

Zejun-Yang commented 6 months ago

Thank you for your valuable suggestions! We believe that using face landmarks may not be the optimal method for controlling face motion, as it could introduce additional noise.