search

hysts / pytorch_mpiigaze

An unofficial PyTorch implementation of MPIIGaze and MPIIFaceGaze
MIT License
346 stars 85 forks source link

problems replacing face detector #40

Closed pperle closed 3 years ago

pperle commented 3 years ago

Thank you for making the project so modular and easy to expand.

Because I have problems with the dlib face detector in non-ideal lighting conditions I would like to replace it with the face detector of mediapipe. Please see my fork.

For some reason the three axis do not align with the 3d head pose. I suspect there is something wrong with my mediapipe landmarks. The rotation matrix of solvePnP looks fine, but maybe the translation vector is off.

Do you have any idea what might be the problem?

This is the output with mediapipe, with all the axis pointing upwards: image

And this is what the landmarks of mediapipe look like: landmarks

hysts commented 3 years ago

Hi, @pperle

I think it's due to the different orientation of the axes, origin, and scale of the 3D model. https://github.com/hysts/pytorch_mpiigaze/blob/master/gaze_estimation/gaze_estimator/common/face_model.py#L12-L28

Applying the following patch should fix the problem.

--- a/gaze_estimation/gaze_estimator/common/face_model.py
+++ b/gaze_estimation/gaze_estimator/common/face_model.py
@@ -580,7 +580,9 @@ class FaceModel:
                 [4.530000, 2.910000, 3.339685]
             ],
                 dtype=np.float)
-            self.LANDMARKS /= 10.0  # cm to mm
+            self.LANDMARKS /= 100  # cm to m
+            self.LANDMARKS -= self.LANDMARKS[1]
+            self.LANDMARKS *= np.array([1, -1, -1])

             dlib_map = [127, 234, 93, 132, 58, 172, 136, 0, 152]
pperle commented 3 years ago

Thank you for your help and the code for the needed 3d model modifications.