search

patrikhuber / eos

A lightweight 3D Morphable Face Model library in modern C++
Apache License 2.0
1.89k stars 596 forks source link

Question about 4DFM #328

Closed Zju-George closed 3 years ago

Zju-George commented 3 years ago

Thanks for sharing the amazing work. I have two questions.

  1. Does this version of eos support 4DFM and if it does, how exactly?
  2. I was amazed by the 36-expression-units feature of 4DFM, but in the video I don't see the opening or closing eye action. Are thoes eye-related action in 36 expression units?
patrikhuber commented 3 years ago

Hi @Zju-George,

Apologies for the late reply, it has been rather busy!

  1. eos supports the 4DFM yes. Most of it can be used out-of-the-box (e.g. all MorphableModel class functionality), if you want to do landmark fitting, then a bit of knowledge about the library and the topic is required to adapt fit-model accordingly.
  2. The latest 4DFM version contains two "blink" blendshapes for the left and right eyes, you might be correct that they're not in that video.

May I ask the background of your questions?

I will close this, as it's not an issue of the eos library - if anything it might fit into the Discussions section. But feel free to reply and ask any follow-up questions, and for commercial inquiries you can also contact 4dface here https://www.4dface.io/contact/.

Zju-George commented 3 years ago

Hi @patrikhuber,

Thank you for your clear explanation!

For background, I actually would like to make some real-time face capture application, especially, using some parametric face model like 4DFM to fit a 3D face to the image.

Yeah, I would also probably consider buying the 4DFM but I hope to make sure first, is the 4DFM just ordinary parametric face model or it is also a differentiable model (something like FLAME model)?

For further commercial inquiries, should I send email to [removed]?

patrikhuber commented 3 years ago

Hi @Zju-George,

I see, thanks for the background!

The 4DFM is as much a "differentiable model" as the FLAME model. It's a linear model, and to generate an instance of the model, only linear operations are needed, which are inherently differentiable. eos returns you the components of the model (mean, basis, eigenvalues etc.) as NumPy array's, and those you can easily put into a PyTorch or TensorFlow tensor. Generating a model instance from coefficients is just a one-liner, which you can then easily express using PyTorch/TensorFlow tensors (here's the C++ code - it's just model_sample = mean + rescaled_pca_basis * alphas). You can easily then use this with a differentiable renderer if you're working with 2D images and need to project points.

For commercial/licensing inquiries, indeed please contact 4dface at https://www.4dface.io/contact/. (I've removed the email address in your post, to prevent spam - also note you had a typo in it, the domain is just .io, without .com.)

Best wishes, Patrik