search

TadasBaltrusaitis / OpenFace

OpenFace – a state-of-the art tool intended for facial landmark detection, head pose estimation, facial action unit recognition, and eye-gaze estimation.
Other
6.84k stars 1.84k forks source link

Landmarks distortion with rotation #935

Open kuzanna2016 opened 3 years ago

kuzanna2016 commented 3 years ago

Hi! Thanks a lot for this project! We are using OpenFace to extract the head pose and eyebrow movement. We decided to calculate the eyebrow raise ourselves and not use the AU option because we need the contours of the movements. So we calculated the distance from the eye-line to brow points. However, we found out that with the head rotation the eyebrow landmarks (and other landmarks too, including the eye points) shift and bend even when there is no movement. See the example below - we took 3 frames from the vertical head rotation video where eyebrows do not move - in the down, centre and top position. We rotated the points using the head rotation angles and centered them on the upper nose point. down_up

Do you have any suggestions on how to eliminate this bias using your code? Our solution now is to use a regression model that predicts the "default" eyebrow distance with the influence of tilting on top of the OpenFace results. Then we use the difference between the calculated "default" distance and the OpenFace "original" distance as our eyebrow movement. I am looking forward to your reply! Thanks in advance!

TadasBaltrusaitis commented 3 years ago

One of the reasons you see this effect is because the shape basis used in OpenFace does not fully disentangle orientation (e.g. some of the PCA components attempt to explain head turning). This means that reconstructed 3D shape is not always going to be accurate and will be affected by rotation up to a point. General 3D reconstruction from a single camera is a really challenging problem, especially when the camera is not constrained (OpenFace was trained on cameras with different focal lengths etc.).

One thing that might make your tracking a bit more stable is by using intrinsic calibration parameters (specifically the focal length fx/fy), as that improves the accuracy of rotation estimation slightly.

Do you have any suggestions on how to eliminate this bias using your code?

Training something that deals explicitly with this bias sounds like a good idea (especially if the bias seems to be consistent across subjects)