Could you please explain why there are two sets of landmarks, innerLandmarkIndex and outerLandmarkIndex

[seanwen86]

Hi, @anhttran I really want to learn all your work. :). And of course there will be many questions for you. :(

In the file "BaselFaceEstimator.cpp", there is the code snippet as below. I couldn't figure out why there are two sets of landmarks, and why the yaw angle could affect the selection of landmark index? Any explanation from anyone would be appreciated. Thanks in advance..

// Get 3D landmarks points (68x3), given 3D shape (Vx3) and yaw angle
cv::Mat BaselFaceEstimator::getLM(cv::Mat shape, float yaw){
    cv::Mat lm(bf.lmInd.rows,3,CV_32F);
    for (int i=0;i<lm.rows;i++){
        int ind;
        if (yaw > 0 && yaw < M_PI/9 && i < 8) ind = bf.lmInd.at<int>(i,0)-1;
        else if (yaw < 0 && yaw > -M_PI/9 && i > 8 && i < 17) ind = bf.lmInd.at<int>(i,0)-1;
        else
        ind = bf.lmInd2.at<int>(i)-1;
        for (int j=0;j<3;j++){
            lm.at<float>(i,j) = shape.at<float>(ind,j);
        }
    }
    return lm;
}

[anhttran]

Hi,

Most of the traditional landmark detectors provide holistic 2D landmarks based on image contour. Hence, the landmark points on face boundary do not correspond to fixed 3D points of the face but vary by the yaw angle.

For example, you can check Dlib landmarks in this image: the boundary points on the left side locates on the front part of the face (inner), while the points on the right side go to under the ear (outer): https://qph.fs.quoracdn.net/main-qimg-391e7898134bbaae1de14f5d14f3f943

I use 2 sets of 3D landmarks to handle this problem. Depends on the yaw angles, I can use all outer 3D landmarks (frontal), inner + outer landmarks for each side (half-profile), or just use outer points for one side and drop the ones on the other side (profile). It is just an approximation but it is cheap and good enough.

In case you have 3D landmark input, like the output of 3DDFA or recent deep learning based methods, you can just use outer landmarks.

[seanwen86]

@anhttran This is a great detailed explanation. Thank you so much. For one landmark, its more precise position should be located between the TWO corresponding landmarks from the TWO 3D landmark sets along the YAW angle. Am I right?