search

facebookresearch / meshtalk

Code for MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement
Other
368 stars 56 forks source link

Issue with Character Facial Orientation and Character Network Topology #47

Closed BarryAlllen closed 1 year ago

BarryAlllen commented 1 year ago

Hi，Thank you very much for your open-source project. Why is the character's orientation not facing forward in the output, and how can I make the character face forward in the generated video? Also, I'd like to ask how to modify the character model without changing the existing network topology. I apologize, as I'm not very familiar with character modeling and related techniques.

alexanderrichard commented 1 year ago

Hi, you can change rotation and translation here: https://github.com/facebookresearch/meshtalk/blob/main/utils/renderer.py#L45

Using other characters is difficult as you need the same topology we use in our meshes. To get more characters, you can use meshes from the multiface dataset which have a compatible topology. There is a script to transform multiface meshes into meshtalk meshes here: https://github.com/facebookresearch/meshtalk/blob/main/utils/multiface2meshtalk.py

BarryAlllen commented 1 year ago

Thank you for your response! I was wondering if you could kindly share what software you use to modify facial modeling and how you adjust the topology. Is it done using MeshLab? Additionally, could you please explain how to individually export data for different facial parts, such as the eyes, mouth, and neck? I'm also curious to know if it's possible to use my own topology along with voice data to train a personalized model. I'm currently a student working in a lab, and my knowledge is limited, so I hope you can understand my inquiries. Thank you for your kind consideration.

alexanderrichard commented 1 year ago

I don't think what you want to do is possible, unless you have a technical artist who can take one of the provided meshes and remodel it to your face. Training a personalized model requires quite some amount of 3D-tracked data of yourself, probably in the range of 1 or 2 hours.

BarryAlllen commented 1 year ago

I appreciate your answer! I also wanted to ask how to modify the appearance of the model while preserving its original topology and what software to use.

alexanderrichard commented 1 year ago

I honestly don't know what's best for your case. This would be a question for an artist who does re-rigging.