search

Rubikplayer / flame-fitting

Example code for the FLAME 3D head model. The code demonstrates how to sample 3D heads from the model, fit the model to 3D keypoints and 3D scans.
http://flame.is.tue.mpg.de/
722 stars 109 forks source link

Increasing the number of points and getting static and dynamic landmark embeddings. #40

Closed filby89 closed 1 year ago

filby89 commented 1 year ago

Hi, thank you very much for your repo! I would like to ask is you have any specific process for getting the static barycentric coordinates, as well as the dynamic landmark embeddings, in case someone wants to add more keypoints.

TimoBolkart commented 1 year ago

To get specific landmark embeddings, we render faces in FLAME topology of different face shapes from varying angles, run the 2D landmark predictors, and project those landmarks on the 3D mesh. Then, we average the landmark locations across the different face shapes, and compute its barycentric embedding. Sharing this pipeline would require some cleanup that I am unable to do right now, but we do have embeddings for instance for around 100 Mediapipe keypoints. We could share this if this would help.

filby89 commented 1 year ago

Thanks, I think I got it! It would be great if you could share the mentioned embeddings for mediapipe kps (and which kps are those). I have generally seen that an increasing number of repos are switching to mediapipe from FAN.

TimoBolkart commented 1 year ago

Makes sense, Mediapipe is more accurate around the eyes and the lips than FAN. You find the landmark embeddings here. I also added them to the FLAME download page.

filby89 commented 1 year ago

Great, thanks again !

ArjunZ786 commented 1 year ago

sorry dumb question, what does FAN stand for?

akasharidas commented 11 months ago

@TimoBolkart The mediapipe embeddings you provided are helpful, but they only contain keypoints around the eyes, nose and lips. I'm trying to add more keypoints especially around the edges of the face or on the contour. If you have code already for the pipeline you mentioned for adding more keypoints, could you please share it?

shunlinlu commented 8 months ago

Hi, @akasharidas

Did you solve it?