search

facebookresearch / meshtalk

Code for MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement
Other
368 stars 56 forks source link

How to train meshtalk on the Multiface dataset? #41

Closed chen8750 closed 1 year ago

chen8750 commented 1 year ago

Thanks for your excellent work! How to use the Multiface dataset with aligned audio and mesh? AND How are the parameters mouth_mask, eye_mask, eye_keypoints obtained?

alexanderrichard commented 1 year ago

Hi, for alignment of audio and meshes, please refer to https://github.com/facebookresearch/multiface/issues/31

To train Meshtalk with Multiface, get the Multiface tracked meshes and audio, write a dataloader, and use the provided code. There are some hints here: https://github.com/facebookresearch/meshtalk/issues/39

Note that Meshtalk has been trained on 250 identities but we were only able to publish 13. I'd recommend to initialize with the pretrained model or use the pretrained model to generate additional training data. Others have found the training to be unstable if you use the 13 identities from Multiface only.

The mouth_mask, eye_mask, and eye_keypoints have been manually selected from the template mesh.