Audio to Mouth-related Motion

YuanxunLu / LiveSpeechPortraits

Live Speech Portraits: Real-Time Photorealistic Talking-Head Animation (SIGGRAPH Asia 2021)

MIT License

1.16k stars 200 forks source link

Audio to Mouth-related Motion #25

Closed TimmmYang closed 2 years ago

TimmmYang commented 2 years ago

Hi, great work! As I read from your paper, one need to get the ground truth mouth displacements to train Audio2Feature model. You mentioned [Shi, 2014; Thies, 2016] to track 3D mouth shape and head pose. Are there any open source tools to help me get these data?

YuanxunLu commented 2 years ago

The used tracking tool can be replaced by any 3D facial tracking algorithms, e.g., 3DDFAv2, PRNet, DECA, etc. There is much open-source work targeting single image 3D face reconstruction.