Landmarks coordinate error

[kangzhiq]

Hello,

Thank you very much for this implementation!

While reviewing the code, I noticed one error on the landmark coordinates. I plot the landmarks stored in the 3D model.

model3D.model_TD

Here is what I got:

For the frontal view, the landmarks seem to be good.

However, if we turn a bit to the bottom view, there are some obvious defects of the landmarks on the right jaw of the model.

Here is a bottom view:

I think these defects would lead to a drop in performance, resulting from a wrongly estimated projection matrix from 3D to 2D.

Could you please correct this error? Thank you in advance！

[dougsouza]

@kangzhiq, this model was provided by the authors. It is possible that something went wrong when loading it using scipy (but that's very unlikely). In their official project page there are instructions on how to generate a new model.

Chers

[TalHassner]

@kangzhiq Thanks for flagging this.

It is a well known problem with using landmarks for face alignment. In fact, it's one of many problems: Even if the points were positioned correctly, keep in mind that facial expression deformations change the physical position of the points leading to alignment errors. Points along the outside of the face capture the face contour, which changes with facial pose, so that introduces even more errors.

Generally speaking, landmarks for face alignment were the best way for us (the community) to align faces before deep learning. These days, there are better ways of aligning faces. You can see our more recent project here which provides a landmark free face alignment method.

[kangzhiq]

@kangzhiq Thanks for flagging this.

It is a well known problem with using landmarks for face alignment. In fact, it's one of many problems: Even if the points were positioned correctly, keep in mind that facial expression deformations change the physical position of the points leading to alignment errors. Points along the outside of the face capture the face contour, which changes with facial pose, so that introduces even more errors.

Generally speaking, landmarks for face alignment were the best way for us (the community) to align faces before deep learning. These days, there are better ways of aligning faces. You can see our more recent project here which provides a landmark free face alignment method.

@TalHassner Thank you for your reply!

I see. So it is kind of a rigid/non-rigid ambiguity between head motions and facial expressions. In fact, it seems strange to me. I manually corrected the landmark positions. Then I found that the frontalization was not better: especially the identity of the person was somehow lost. However, with the default setting, the identity of the person is well preserved.

Anyway, I am currently working on frontalization research, so your work is very valuable to me. Thank you again and I will surely follow your other publications. :-)

[TalHassner]

Oh, the real thanks should go to @dougsouza for making this excellent Python version available!

On face alignment, I think the important thing to remember is that in 2020, you no longer need local processing for landmark detection and can get to the alignment transformation directly by regressing it using a network.

I mentioned our FacePoseNet paper but you can also look at its follow up paper which provides a lot more evaluations or our paper on how different 3D face alignments can easily be generated on the fly for face recognition. They all have code :)

All the best!