3D landmarks wrt camera not the same as rotated and translated landmarks wrt object

[crisie]

Hi Tadas,

For my application I need to fit a 3D object to the head and for that I need to compute the center of the head (not the center of the face). To do so, I thought of using the 3D landmarks in object space (clnf_model.pdm.CalcShape3D...) to compute the head center - roughly 10 cm behind landmark 28 - and then apply Rotation and Translation computed with GetCorrectedPoseWorld to such landmarks to find the point in world coordinates. However, I've seen that the 3D landmarks computed this way do not match the 3D camera landmarks obtained with clnfmodel.GetShape...; X and Y are almost the same but Z has a deviation - the rotated and translated landmarks wrt object are always further from the camera than the 3D camera landmarks.

I've checked the code and seen the way of computing the position of the landmarks changes slightly among methods. Which method is more reliable for obtaining the real 3D landmarks and thus finding a correct head center?

Thanks a lot, and also congratulations on your work!

[TadasBaltrusaitis]

The reason for this is because the landmark detection module assumes an orthographic (and not a perspective) camera model as it is easier to fit. While the GetCorrectedPoseWorld and GetShape methods actually model a perspective camera and attempt to correct for it. If you project the landmarks acquired by GetShape method onto the 2D image using a perspective camera, you will notice them matching the 2D detected landmarks.

If you have a decent estimate of intrinsic camera parameters (fx, fy, cx, cy) I would suggest you use the perspective aware methods.

I would also suggest you have a look at the DrawBox method which does a reasonably good job at drawing a 3D bounding box around the head rather than the face and it does so in 3D space and then projects the bounding box.