face landmark detection without dlib

[anguoyang]

Hi @ZhenglinZhou , thank you for the contribution, I found that the demo.py used dlib for landmark detection, just wonder if I could do the landmark detection without dlib, is it possible to get the landmark with STAR only?

[ZhenglinZhou]

Hi @anguoyang , thanks for your interest!

Yes. STAR is a facial landmark detector, which is a downstream task of face detection. In demo.py, we use dlib to get face bbox. If you want to use STAR without dlib, I suggest training a new STAR with the other face detector you want to align the preprocess for better performance.

[anguoyang]

thank you so much for the clarification, if dlib is just for face detection to get bbox, then it is fine:) I just worried if it was for extracting landmarks or other additional information before using STAR.

[anguoyang]

@ZhenglinZhou sorry for asking again,I found that in your demo code, the shape predictor also be used to get some values, like scale, center_w, center_h... , I don't know what does it mean, especially, why used 200 and 1.05 to get the scale? could I calculate the scale without dlib/sp?

 for i in range(68):
        x = face.part(i).x
        y = face.part(i).y
        shape.append((x, y))
    shape = np.array(shape)
    # image_draw = draw_pts(image_draw, shape)
    x1, x2 = shape[:, 0].min(), shape[:, 0].max()
    y1, y2 = shape[:, 1].min(), shape[:, 1].max()
    scale = min(x2 - x1, y2 - y1) / 200 * 1.05
    center_w = (x2 + x1) / 2
    center_h = (y2 + y1) / 2 

[ZhenglinZhou]

Hi @anguoyang, feel free to leave a comment!

The center_w and center_h are the centric point of the face box. More details can be found here.

To calculate the scale, 200 is a normalized content, which will not effect the preprocess. Meanwhile, the 1.05 is a trick, which expand the face box outward by 0.05. If you use other face detectors, it can be initially set to 1.0.

For other face detector, I suggest training a new STAR to make sure the preprocess is aligned.

[anguoyang]

Hi @ZhenglinZhou , thank you fo the clarification, If I cropped and saved the face ROI with rectangle of [detection in dets], the result image seems normal(although still different with other detectors): https://ibb.co/gFNVt9D however, if with x1,y1,x2,y2, the part above eyebrow has been cut out, like this: https://ibb.co/j6SK79d

and if I replaced with other detectors, the landmarks are incorrect due to the wrong scale, is there possibility to use other detectors, like retinaface to get bbox, to get the correct landmarks without re-training?

[ZhenglinZhou]

Hi @anguoyang, I get it!

You could try to replace the detector in demo.py with your own face detector and do not change other setting.

If it does not work, you might be considering to retrain the STAR to make sure the preprocess is aligned.

[anguoyang]

Hi @ZhenglinZhou , I already tried with other detector:), yes it doesn't work, the output is incorrect, I didn't use sp to get x1,x2,y1,y2, as I don't want to use dlib, I just used my bbox to calculate the scale/center directly. the landmarks/mesh shrinked into a very small size.

[ZhenglinZhou]

Hi @anguoyang.

I think STAR might be sensitive to the preprocess. If the preprocess is not aligned, the performance will be hurt. You might need to retrain the STAR with your own face detector.