search

TimoBolkart / TF_FLAME

Tensorflow framework for the FLAME 3D head model. The code demonstrates how to sample 3D heads from the model, fit the model to 2D or 3D keypoints, and how to generate textured head meshes from Images.
http://flame.is.tue.mpg.de/
444 stars 78 forks source link

Adding More Landmarks #30

Closed alexrichardson21 closed 4 years ago

alexrichardson21 commented 4 years ago

I've noticed the neck tends to be a little off because the flame landmark embedding is missing any sort of neck landmark. If I wanted to add to the embedding using some openpose landmarks, how would i find the barycentric coordinates for the new landmarks?

alexrichardson21 commented 4 years ago

Or is there a way to build a more cohesive temporal fitting process instead of just the single frame?

TimoBolkart commented 4 years ago

You are right, the difficulty is that for face-only images, no good landmark predictor exists to constrain the neck pose (OpenPose also does a poor job for these cases). If you have a fullbody image, you can actually fit SMPL-X (expressive body model) instead of FLAME (head only) using SMPLify-X. If you are only interested in the head, you can then still crop the head-only vertices (which are basically the FLAME vertices). Howerver this will give you a much better neck.

TimoBolkart commented 4 years ago

Another option could be not only optimizing a landmark loss but additionally optimizing some photometric loss (see e.g. this demo for details how to do this). This provides more supervision than keypoints only. And yes, certainly one could do a joint optimization with some temporal regularization to multiple consecutive frames of a video.

alexrichardson21 commented 4 years ago

Interesting. I was using SMPLify X before but it runs significantly slower than FLAME fitting. Haven't looked into photometric loss yet so that seems worth a try

hbinol commented 4 years ago

Interesting. I was using SMPLify X before but it runs significantly slower than FLAME fitting. Haven't looked into photometric loss yet so that seems worth a try

Hi Alex!

I wonder if you had a chance to look at any neck landmarks/positions?