search

Rubikplayer / flame-fitting

Example code for the FLAME 3D head model. The code demonstrates how to sample 3D heads from the model, fit the model to 3D keypoints and 3D scans.
http://flame.is.tue.mpg.de/
741 stars 111 forks source link

Can FLAME estimate eyeball rotation when only using landmarks for fitting? #33

Open hengfei-wang opened 1 year ago

hengfei-wang commented 1 year ago

Flame has the parameters which can control eyeball rotation. But I saw that the landmark used for flame fitting doesn't include pupil center which may make it impossible to estimate eyeball rotation parameters only from landmarks?

TimoBolkart commented 1 year ago

You are right, estimating eye gaze by optimizing FLAME's eyeball pose parameters requires additional landmarks for the pupil center. If you are using a landmark predictor that provides pupil center locations, one can easily add the distance between FLAME's projected pupil center and the predicted pupil locations (i.e., same as the existing landmark loss but for an additional pair of points) as additional loss and optimize for FLAME's eyeball parameters.

hengfei-wang commented 1 year ago

Thank you! I will give it a try.

ArjunZ786 commented 1 year ago

I would like to do something similar not for eyes but like add a few more known landmarks. Were you able to add it? Is so can you share steps or code to do the same?