search

Rubikplayer / flame-fitting

Example code for the FLAME 3D head model. The code demonstrates how to sample 3D heads from the model, fit the model to 3D keypoints and 3D scans.
http://flame.is.tue.mpg.de/
694 stars 107 forks source link

Can flame exactly reconstruct 3D human face shape with only 51 landmarks？ #36

Open DatanIMU opened 1 year ago

DatanIMU commented 1 year ago

51 landmarks don't include face shape, Can flame exactly reconstruct 3D human face shape with only 51 landmarks？or just do a rough calculation with face shape.

in fact， it's possible that two human faces have the same nose, eyes, mouth，and brow， but one is fat and the other is thin.

TimoBolkart commented 1 year ago

You are right, the placement of the 51 landmarks is designed to capture expressions not shape. FLAME can capture a large variation of face shapes, but not from landmarks alone. The landmark fitting demo is there to demonstrate how to use FLAME, not how to get the best 3D reconstruction from a single image. For this there are other tools available. For an overview of tools that use FLAME see the FLAME-Universe.

In practice, one would use additional terms like a photometric loss optimized with differentiable rendering (see a demo here). If one is interested in good 3D shapes from images, I would instead go for one of the available regressors like MICA which regresses metrically accurate 3D faces directly from images. Alternative, check out tools like DECA. Interestingly, landmarks alone also don't capture all variation in expression (i.e., frowing is not captured well, and so are some extreme expressions of emotions). For getting good emotional faces from images, one can for instance use EMOCA.

DatanIMU commented 1 year ago

thanks for your kindly reply and suggestion. I want to use MICa to do face reconstruction, and I can get 300 flame morph keys, I can view it in blender, but it seems not cool enough. I want to do reconstruction in daz3d or metahuman use mica. however they have different morph keys and vertex. If I still want to use mica, I must find the relationship between flame and daz metahuman. Yestoday I think if I can landmark metahuman and use flame fitting to get flame keys, and do bp ai train between metahuman keys and flame keys, then I can reconstruction face by a picture use mica to get 300 flame keys, and then get metahuman keys through bp. Now I know flame fitting can not help me, because 51 landmark is not paticularity. I notice there is a paper transfer BFM to flame, is there a tool can transfer any unreal engine model like daz3d or metahuman to flame. Please show me a way to do it

shunlinlu commented 5 months ago

Hi, @maemoKKK

Did you find a way to solve it?

Shunlin