Significant drop of the accuracy for tilted face

[Jendker]

I am testing the different images and I experience significant drop of accuracy if I just feed the tilted face with >30*.

I am using the prepared examples in browser and node-js and effects are similar: for the face looking straight into camera I get around 0.3 - 0.4, but if I tilt the face I go beyond 0.7. When testing the same images with dlib (which is the original network used in face-api.js if I understand correctly) I get the distance of <0.5 for the tilted faces.

Do I need to additionally enable the alignment with face-api.js? Maybe I am missing something here.

Here the example photos, where I get high distance with face-api.js, but is accepted by dlib:

But if I just manually rotate further the first photo:

the face is accepted and I get very similar result to dlib (around 0.48).

[justadudewhohacks]

Thanks for reporting this! The issue is, that face-api.js alignment doesn't rotate the images, it only centers them at the moment. I guess dlib does rotate them based on the face landmarks predicted by the shape predictor right?

[Jendker]

Yes, I am not sure how exactly is it done to mimic it, but it is done here: http://dlib.net/dnn_face_recognition_ex.cpp.html in the functionextract_image_chip

Here even 5 landmark model is used instead of 68 point model. Maybe this 5 point model after quantization would be even smaller than the currently used model in face-api.js for face recognition :)

I could try to make a PR with it, but I am normally writing in C++, someone would need to review it, thoroughly.

[justadudewhohacks]

Maybe in future it would be beneficial to train a very small 5 point landmark model just for alignment, although the existing 68 point face landmark models should be really fast already, so not sure if thats worth it.

A PR would be highly appreciated. Currently the alignment is done based on the 68 point face landmarks and is simply centering the face by some predefined ratio, that is close to how dlib does it. The rotation of the face is not considered yet, I think the main challenge would be to figure out, how to rotate the image content of an HTML canvas or image: FaceLandmarks.ts

[Jendker]

Thanks a lot! I'll try to come back with PR if I find some time for it.

[justadudewhohacks]

Just gonna leave this issue open to keep track of it.

[Bajajar]

Hi Jendker,

You can rotate the image by using landmark position at 36 (left eye) and 45 (right eye) as begin and end point then you calculate the angle of the line compared with horizontal.

For example: https://github.com/justadudewhohacks/face-api.js/blob/master/examples/examples-browser/views/bbtFaceLandmarkDetection.html

at line 42:

function redraw() { const canvas = faceapi.createCanvasFromMedia(currentImg) $('#faceContainer').empty() $('#faceContainer').append(canvas) faceapi.drawLandmarks(canvas, landmarks, { lineWidth: drawLines ? 2 : 4, drawLines }) }

You can edit as

function redraw() { var canvas = faceapi.createCanvasFromMedia(currentImg) var dY = landmarks['positions'][45]['y'] - landmarks['positions'][36]['y'] var dX = landmarks['positions'][45]['x'] - landmarks['positions'][36]['x'] var degree = Math.atan(dY/dX) var ctx = canvas.getContext("2d"); ctx.rotate(-degree); ctx.drawImage(currentImg,0,0); $('#faceContainer').empty() $('#faceContainer').append(canvas) faceapi.drawLandmarks(canvas, landmarks, { lineWidth: drawLines ? 2 : 4, drawLines }) }

Hope I can help.

[Jendker]

That could be something, thanks!

I was thinking about a bit different approach with the alignment to follow what is done in dlib. Davisking is using the affine transformations with some approximations, which are quite complex and if we are using the original dlib face recognition model it would be advisable to follow it to get better recognition results. On the other hand it works still well if we are not using any transformation just image cropping :) That could be an intermediate solution.

The other thing is how to add it neatly into face-api.js.

[Cayan]

Hello everyone, I'm trying to use this API on my project to detect faces but I've noticed that I have a lot of different users that try to submit a photo with their faces tilted. Has been there any update on this particular issue?

[Jendker]

I ended up using a solution recommended by Bajajar and it was fine for me. I didn't manage to get it working by changing the underlying face-api.js functions, my limited JavaScript skills were not sufficient.

[Cayan]

@Jendker thank you for your quick respose!

I've noticed that the recognition API doesn't work well with tilted images but the detection is doing great!

Thank you.

[wkdhkr]

https://github.com/Jack-CV/FaceKit/tree/master/PCN

If face-api.js uses such a model, it will be able to recognize faces of all angles. Face-api.js can not meet my requirements because the landmarks are messy and unmatched on a lying face or a face that is flipped up and down.

[justadudewhohacks]

@wkdhkr thanks for sharing the repo, I will look into the paper and if it looks promising.

The landmark model is not trained on data of faces that are rotated like 90 degrees. You could simply rotate the image in 90 degree steps and feed them into the model, but that obviously requires more computation time.

[wkdhkr]

@justadudewhohacks

https://github.com/siriusdemon/pytorch-PCN There is a Pytorch version of the above model. If this is converted into a model for Tensorflow.js with ONNX.js etc., it seems that it can be used with face-api.js. (I do not understand the problem of license well.) Alternatively, it might be possible to bridge the C implementation as a native module.

By the way, Not only landmarks but also ssd mobile net can not detect inverted or rotated faces. Rather, if it is possible to detect the top and bottom of the face first, the landmark model as it is may be fine even with the current landmark model. Because processing only needs to rotate the face once.

[Liel208]

Any new updates on this problem?

[karaposu]

Hi Jendker,

You can rotate the image by using landmark position at 36 (left eye) and 45 (right eye) as begin and end point then you calculate the angle of the line compared with horizontal.

For example: https://github.com/justadudewhohacks/face-api.js/blob/master/examples/examples-browser/views/bbtFaceLandmarkDetection.html

at line 42:

function redraw() { const canvas = faceapi.createCanvasFromMedia(currentImg) $('#faceContainer').empty() $('#faceContainer').append(canvas) faceapi.drawLandmarks(canvas, landmarks, { lineWidth: drawLines ? 2 : 4, drawLines }) }

You can edit as

function redraw() { var canvas = faceapi.createCanvasFromMedia(currentImg) var dY = landmarks['positions'][45]['y'] - landmarks['positions'][36]['y'] var dX = landmarks['positions'][45]['x'] - landmarks['positions'][36]['x'] var degree = Math.atan(dY/dX) var ctx = canvas.getContext("2d"); ctx.rotate(-degree); ctx.drawImage(currentImg,0,0); $('#faceContainer').empty() $('#faceContainer').append(canvas) faceapi.drawLandmarks(canvas, landmarks, { lineWidth: drawLines ? 2 : 4, drawLines }) }

Hope I can help.

landmark accuracy drops if the face is tilted. So I don't understand how can we trust 36 and 45 in this case ?