search

radekd91 / emoca

Official repository accompanying a CVPR 2022 paper EMOCA: Emotion Driven Monocular Face Capture And Animation. EMOCA takes a single image of a face as input and produces a 3D reconstruction. EMOCA sets the new standard on reconstructing highly emotional images in-the-wild
https://emoca.is.tue.mpg.de/
Other
689 stars 89 forks source link

Not good for wink expression #16

Open wwdok opened 2 years ago

wwdok commented 2 years ago

I test EMOCA with some wink expression images, but results seems not good : wink_girl inputs geometry_detail

wink_ironman geometry_detail inputs

wink_obama geometry_detail inputs

Do you have some ideas to improve the result ?

radekd91 commented 2 years ago

Sadly, this is true. As it so happens, winks like these are not very well represented in the AffectNet dataset as they don't necessarily correspond into one of the basic expressions (happy, sad, angry, ...). As a consequence, the emotion network is not trained to recognize these and therefore it does not supervise EMOCA to reconstruct it well.

A better, more finegrained emotion dataset would help alleviate this limitation. The emotion network would have to be retrained as well as EMOCA. I'm afraid there is no easy way to hack this in code-wise. It's more of a data issue.

radekd91 commented 2 years ago

A small update, EMOCA itself and the AffectNet dataset may actually have some wink-like expressions. I have been informed, that inability to create wink shapes might run deeper than that - it seems that the FLAME face model's training data did not really contain any winks and therefore it would not be able to generate these very well.

radekd91 commented 1 year ago

Update, @wwdok , I recommend giving the newly release EMOCA 2.0 a try. It will most likely do a lot better. Thanks again for your feedback