search

patrikhuber / eos

A lightweight 3D Morphable Face Model library in modern C++
Apache License 2.0
1.89k stars 596 forks source link

Could the input image be gray or NIR image? #323

Closed briantse100 closed 3 years ago

briantse100 commented 3 years ago

Hi, @patrikhuber I have run the sample with RGB image. And the result is good. But the input is gray or NIR image, it is error. I have known the 3D face model is built on RGB image. So is it only suitable for RGB image. Or how to modify the code for the gray or NIR image. Thanks!

patrikhuber commented 3 years ago

Hi,

eos's main fitting (and the one that the fit-model example uses) is shape-to-landmarks fitting - no image information beyond the given facial landmarks are used. The landmark fitting works on whatever landmarks you give it. If you have NIR images, you probably want to use a landmark detector that's trained on NIR images. If the accuracy of landmarks-only fitting is not suitable for you, then you may want to look into non-linear/analysis-by-synthesis fitting (see fit-model-ceres and lots of literature about 3D face model fitting out there). Indeed if you want to use a 3DMM colour model in that scenario and with NIR images, you may need to make some adjustments and/or ideally have a colour model learned in NIR-space. There's plenty of options. There are also related (open & closed) GitHub issues discussing this. I would advise you to get more familiar with the topic and the literature.