[Example] Face landmark detection

[Wenzhao-Xiang]

I did some investigation for face landmark detection. It seems not so good to implement an end-to-end model, since it's difficult for one model to detect face feature from a whole image.

So, plan to separate this task to two steps with two models:

1. Face detection for input image
2. For each face, detect its key points

[huningxin]

There are some potential references:

1. DeepFace
2. FaceNet
3. Face detection and landmarks detection models of OpenVINO model zoo
4. Face detection and recognition models of ONNX model zoo
5. OpenFace used by OpenCV.js dnn example

[Wenzhao-Xiang]

Nice references! Thanks!

[Wenzhao-Xiang]

For face detection, there are three popular models now:

1. SSD
2. YOLO
3. MTCNN The first two models are migrated from object detection with only one class "face". And the third model is used for face detection and alignment, which can output both detected face and five key points.

For worrying of performance on web, first I will have a try with Tiny-YOLO-v2 for face detection.

[Wenzhao-Xiang]

1. I implemented tiny-yolo-v2 for face detection Reference: YoloKerasFaceDetection

2. This model is migrated from Tiny-YOLO-V2 object detection model. The output of face detection model is reduced to [13,13,30], because there is only one class face in label list.(The original model's output is [13,13,125] with 20 classes)

3. Tiny-YOLO-V2 uses new operation LeakyRelu . But when convert keras model to tflite model, LeakyRelu is replaced by MUL+Maximum, like this: So new op we need is Maximum.

4. A simple image and video demo with WebGL backend. input size [416, 416, 3] => output size [13, 13, 30] GPU: 1080TI

Single face:

Multiply face:

Tiny-yolo-V2 is fast but with low accuracy when multiply face. I will try other models later.

[Wenzhao-Xiang]

Visit https://wenzhao-xiang.github.io/intel/webml-examples/index.html for face detection demo.

[Wenzhao-Xiang]

A face detection video demo, about 29~30 FPS with 1080Ti

[Wenzhao-Xiang]

1. I implemented a CNN modelo for face alignment based on face detection(tiny_yolo_v2) above. Reference: cnn-facial-landmark

2. This model is just a simple CNN model with the following ops:

   The output contains 68 key points like this:

3. This model strongly depends on the accuracy of face detection. Seems tiny-yolo-v2 can't meet its needs. Now the video demo is more OK, but image demo usually fails when comes to irregular image inputs. Two solution : (1) Look for better face detection model (2) Look for better face alignment model

4. A simple image and video demo with WebGL backend. input size [128, 128, 3] => output size [1, 128] (68 key points) GPU: 1080TI

Success samples:

Image demo:

Video demo:(about 22 FPS)

Fail samples:

The samples suggest that this model only works well with boxes closed to the square.

I'll be going on this example.

[Wenzhao-Xiang]

I investigated MTCNN last week, but it seems that MTCNN isn't suitable for our API. MTCNN models have three stages.

1. Stage 1: Proposal Net, provides a large number of candidate windows, computes the bounding box regression vector and reduces the window with NMS.
2. Stage 2: Refining Net, rejects some non-face windows, calculates bounding box regression vectors and reduces the window with NMS.
3. Stage 3: Outputting Net, output 5 landmark points on the face.

The first stage need different scaled input images, which means an uncertain input tensor for the first network. However, our API need all of the tensors to be certain when initializing, conflicting with MTCNN.

[huningxin]

conflicting with MTCNN.

Thanks for the investigation. I am wondering how MTCNN implements by using DL frameworks. Do you have any ideas?

[Wenzhao-Xiang]

@huningxin Other DL frameworks like keras support a dynamic input. For example,

None can be arbitrary value. But our API allocate all the tensor when constructing graph, right?

[Wenzhao-Xiang]

After use SSD model for face detector, landmark detection result work better now! The face detection models are SSD_MobilenetV1, SSD_MobilenetV2, SSDLite_MobilenetV2 and Tiny_YoloV2. And face alignment model is DAN(Deep Alignment Network, here is the paper).

I have pushed my code to my github io, please go to https://wenzhao-xiang.github.io/intel/webml-examples/index.html for a simple face detection and alignment image/camera demo.

Single face:

Multiple face: