search

Doubiiu / CodeTalker

[CVPR 2023] CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior
MIT License
515 stars 56 forks source link

Can you animate any FLAME face model like voca does #45

Open GrizzCMX opened 1 year ago

GrizzCMX commented 1 year ago

Can I use the flame face model outside the vocaset(Training set, Validation set and test set) as a template to generate face animation, how do I set it up, and can I do it using a pre-trained model? Or need to retrain the model

Doubiiu commented 1 year ago

You can check the templates.pkl file and define your own templates.pkl to use new templates. Sure you can use the pre-trained model.

GrizzCMX commented 1 year ago

Thanks for your reply! I will try it agin

LeoniusChen commented 1 year ago

You can check the templates.pkl file and define your own templates.pkl to use new templates. Sure you can use the pre-trained model.

Hi, do you know how to get a templates.pkl from a single portrait image? Any open-source repo to be found?

Doubiiu commented 1 year ago

@LeoniusChen Hi, DECA is a widely used 3D face reconstruction method by estimating FLAME parameters, and then they can be converted to mesh representations. Please check the FLAME repo for more info.

LeoniusChen commented 1 year ago

@LeoniusChen Hi, DECA is a widely used 3D face reconstruction method by estimating FLAME parameters, and then they can be converted to mesh representations. Please check the FLAME repo for more info.

I successfully obtained the template using DECA and FLAME, and replace the template in the codes https://github.com/Doubiiu/CodeTalker/blob/1a65bfe2e9e164830500d0ccdf5bb55e9f9898c3/main/demo.py#L179 and https://github.com/Doubiiu/CodeTalker/blob/1a65bfe2e9e164830500d0ccdf5bb55e9f9898c3/main/demo.py#L136 . However, the result is not satisfying. I wonder if the subject for inference should be carefully chosen?

https://github.com/Doubiiu/CodeTalker/assets/45724515/10c43c37-0360-4d42-86dd-5aede168a66a

Doubiiu commented 1 year ago

Hi, 'subject' is used to change the template for displaying, since you have utilized your own subject, there should be no effect when you are changing 'subject' in demo.yaml. But I think chaning 'condition' in the config may influence the results, you can have a try. BTW, the unsatisfactory results may be caused by domain gap between Chinese and English. You may check if the English audio can work well.

zk19971101 commented 3 months ago

@Doubiiu @LeoniusChen Is the color and context information from a single portrait image can be add to tamplate or Mesh?

hrWong commented 1 month ago

Hi @Doubiiu @LeoniusChen I'm using DECA and can't get the .pkl template, I'd like to ask you how exactly did you get the template.pkl?