search

EvelynFan / FaceFormer

[CVPR 2022] FaceFormer: Speech-Driven 3D Facial Animation with Transformers
MIT License
778 stars 133 forks source link

How to FaceFormer output arkit blendshape coefficient instead of vertices? #35

Open youngstu opened 2 years ago

youngstu commented 2 years ago

The work is remarkable. How to FaceFormer output arkit blendshape coefficient instead of vertices? Very Thanks.

youngstu commented 2 years ago

If faceformer supports arkit-blendshape-coefficient output, it can support other 3D templates.

lucasjinreal commented 1 year ago

@youngstu Did u get a way to do this? I got some problem

EvelynFan commented 1 year ago

The work is remarkable. How to FaceFormer output arkit blendshape coefficient instead of vertices? Very Thanks.

Hi, for the current model, the output representation is vertex positions. The training datasets we used do not contain the blendshape coefficients ground truth, so the model was not able to learn such information during training. I guess some other techniques based on inverse graphics may be required to convert the vertex positions into 51 blendshape coefficients.

zhaiyuan0217 commented 1 year ago

The work is remarkable. How to FaceFormer output arkit blendshape coefficient instead of vertices? Very Thanks.

Hi, for the current model, the output representation is vertex positions. The training datasets we used do not contain the blendshape coefficients ground truth, so the model was not able to learn such information during training. I guess some other techniques based on inverse graphics may be required to convert the vertex positions into 51 blendshape coefficients.

I want to know if i have the blendshape coefficients groud truth, why in the render video, the lip animation is not evident

zhaiyuan0217 commented 1 year ago

@youngstu Did u get a way to do this? I got some problem

what's problem did you encounter, and did you solve it

lucasjinreal commented 1 year ago

@zhaiyuan0217 not yet, did u able to get 51 blendshapes ?

zhaiyuan0217 commented 1 year ago

@zhaiyuan0217 not yet, did u able to get 51 blendshapes ?

we have blendshapes with our data, after traing the model with our data and blendshapes, the rendered video is not well, but traing the model with vertices, the rendered video is well.

lucasjinreal commented 1 year ago

@zhaiyuan0217 Hi, what do u mean training the model with vertices

zhaiyuan0217 commented 1 year ago

i mean the output is 5023*3 vertices.

xiaodongyichuan commented 1 year ago

@zhaiyuan0217 not yet, did u able to get 51 blendshapes ?

we have blendshapes with our data, after traing the model with our data and blendshapes, the rendered video is not well, but traing the model with vertices, the rendered video is well.

i have same question

Shirley-0708 commented 1 year ago

May I ask how you create your own dataset? Can you be specific? Thank you！ @zhaiyuan0217 @xiaodongyichuan

zhiyoujingtian commented 1 year ago

@zhaiyuan0217 not yet, did u able to get 51 blendshapes ?

we have blendshapes with our data, after traing the model with our data and blendshapes, the rendered video is not well, but traing the model with vertices, the rendered video is well.

Did you solve this problem? I also trained with wav and blendshape data, and the final driving effect is that the mouth basically does not open

agupta54 commented 5 months ago

I was trying to train on DECA expressions but the mouth is not moving when rendering the result.