patrikhuber / eos

A lightweight 3D Morphable Face Model library in modern C++
Apache License 2.0
1.92k stars 598 forks source link

Understanding the fitting algorithm #92

Closed lwyclara closed 7 years ago

lwyclara commented 7 years ago

Hi, thank you for your contribution. I enjoy this repository very much :) I'm trying to understand the algorithm in fitting::fit_shape_and_pose better. Does the surrey face model provide a 1-to-1 mapping to the ibug contour points? If so, can we simply use those sets of contour points for fitting? I'm guessing the special handling of front-facing contour points/ occluding edges currently used provide better accuracy. The handling of contour points is not documented in [1] Fitting 3D Morphable Models using Local Features, or [2] A Multiresolution 3D Morphable Face Model and Fitting Framework. Are there other resources that can help me better understand the method?

patrikhuber commented 7 years ago

Hi, thank you very much!

Does the surrey face model provide a 1-to-1 mapping to the ibug contour points?

Only for the front-facing contour - see model_contours.json. The face contour that is facing away is fitted using occluding-contours, similarly to the algorithm of the Smith 2016 paper. There should be some documentation in the respective files/functions. The front-facing contour fitting is described in this paper. The away-facing contour fitting should be described sufficiently in the function documentation and Smith 2016 paper.

Yes, they both provide way better accuracy! :-) It also helps if you step through the code in a debugger (you can step over irrelevant sub-functions). I think the code should be fairly well documented, if some place is not, please point to a specific instance. We hope to eventually have a paper that describes everything in one place but keep in mind the code evolves quite quick, much quicker than any paper could - so the code documentation is always the best place to look.

lwyclara commented 7 years ago

Thank you! I'll go check out those papers!