Crop the faces for analysis changes the result of the descriptors

[matiasdelellis]

Hi @stalker314314

I had this test quite ago but now polishes it a bit ..

To investigate: I simply reproduced the crop and analysis code, and compared it with not cropping the the same image.

The interesting thing, is both methods have a distance of 0.11836271306919 ¿¿¡!??? :confused: And we are discussing between using 0.5 or 0.4 sensitivity.. :sweat_smile:

Test:

[matias@nube facerecognition]# sudo -u apache php test.php 
We will calculate the descriptors in the same way that NC Facerecognition does... Done.
Now we will calculate the descriptors with default pdlib method... Done.
The distance between both methods is: 0.11836271306919

Source code:

Test Source code:

``` detect($img_path); foreach($facesFound as $faceFound) { print ("We will calculate the descriptors in the same way that NC Facerecognition does... "); // Get rect to crop image. // Source: https://github.com/matiasdelellis/facerecognition/blob/c41bf9c46bd479c28dea8d751c5923d18b0493a9/lib/Db/Face.php#L114 $left = max($faceFound["left"], 0); $right = $faceFound["right"]; $top = max($faceFound["top"], 0); $bottom = $faceFound["bottom"]; $width = $right - $left; $height = $bottom - $top; // Crop image. // Source: https://github.com/matiasdelellis/facerecognition/blob/master/lib/BackgroundJob/Tasks/ImageProcessingTask.php#L335 $image = imagecreatefromjpeg($img_path); $imageCrop = imagecrop($image, [ 'x' => (int)$left, 'y' => (int)$top, 'width' => (int)$width, 'height' => (int)$height ]); imagejpeg($imageCrop, $img_crop_path, 100); // Get landmarks amd compute descriptors // Source: https://github.com/matiasdelellis/facerecognition/blob/master/lib/BackgroundJob/Tasks/ImageProcessingTask.php#L323 $fld = new FaceLandmarkDetection("vendor/models/1/shape_predictor_5_face_landmarks.dat"); $fr = new FaceRecognition("vendor/models/1/dlib_face_recognition_resnet_model_v1.dat"); $landmarks = $fld->detect($img_crop_path, array( "left" => 0, "top" => 0, "bottom" => (int)$height, "right" => (int)$width )); $descriptor = $fr->computeDescriptor($img_crop_path, $landmarks); print ("Done.\n"); print ("Now we will calculate the descriptors with default pdlib method... "); // pdlib method. // Source: https://github.com/goodspb/pdlib#general-usage $fld = new FaceLandmarkDetection("vendor/models/1/shape_predictor_5_face_landmarks.dat"); $full_landmarks = $fld->detect($img_path, $faceFound); $descriptorAligned = $fr->computeDescriptor($img_path, $full_landmarks); print ("Done.\n"); // Calculate distance. // Source: https://github.com/matiasdelellis/facerecognition/blob/master/lib/Helper/Euclidean.php#L32 $n = count($descriptor); $sum = 0; for ($i = 0; $i < $n; $i++) { $sum += ($descriptorAligned[$i] - $descriptor[$i]) * ($descriptorAligned[$i] - $descriptor[$i]); } print("The distance between both methods is: " . sqrt($sum) . "\n"); } ```

Test image:

Croped Face

Resume

I do not say that one is better than another, only that they are different, and I check this with many photos.

[matiasdelellis]

I suspected that could be this, and now I guess it's important..

See: https://github.com/ageitgey/face_recognition/issues/33

Ever talk to you about the alignment of the faces, and the importance of it, however dlib does it automatically. In short, dlib first align and center the faces, and then cut them out with a margin of 25%. This margin adds the chin and more front in the example image, which can infer more information for the analysis.

So, by cutting the image with the landmaks, dlib can not expand the image to get more information. :open_mouth:

[matiasdelellis]

I'm playing with the same test adding the margin and compare..

Source code:

Test Source code:

``` detect($img_path); foreach($facesFound as $faceFound) { // Get image size as limits to crop. $image = ImageCreateFromJpeg($img_path); $iWidth = imagesx($image); $iHeight = imagesy($image); print ("We will calculate the descriptors with default pdlib method... "); // pdlib method. // Source: https://github.com/goodspb/pdlib#general-usage $fld = new FaceLandmarkDetection("vendor/models/1/shape_predictor_5_face_landmarks.dat"); $fr = new FaceRecognition("vendor/models/1/dlib_face_recognition_resnet_model_v1.dat"); $full_landmarks = $fld->detect($img_path, $faceFound); $descriptorAligned = $fr->computeDescriptor($img_path, $full_landmarks); print ("Done.\n"); print ("Now we will calculate the descriptors in the same way that NC Facerecognition does with diferent margins.\n"); // Get rect to crop image. // Source: https://github.com/matiasdelellis/facerecognition/blob/c41bf9c46bd479c28dea8d751c5923d18b0493a9/lib/Db/Face.php#L114 $oLeft = $faceFound["left"]; $oRight = $faceFound["right"]; $oTop = $faceFound["top"]; $oBottom = $faceFound["bottom"]; // Face size.. $fWidth = (int)($oRight - $oLeft); $fHeight = (int)($oBottom - $oTop); // We print a csv table.. print("left; top; right; bottom; percent; distance\n"); // We apply a padding of diferents percents betwen 0 to 50% for ($percent = 0 ; $percent <= 200 ; $percent += 1) { // Calcultate rectangular padding. $wPadding = (int)(($fWidth*($percent/100)/2)); $hPadding = (int)(($fHeight*($percent/100)/2)); // Calculate the real margins, according to the size of the image and face position. $lPadding = min($wPadding, $oLeft); $rPadding = min($wPadding, ($iWidth - $oLeft)); $tPadding = min($hPadding, $oTop); $bPadding = min($hPadding, ($iHeight - $oBottom)); // Calculate new rect to crop image. $left = $oLeft - $lPadding; $right = $oRight + $rPadding; $top = $oTop - $tPadding; $bottom = $oBottom + $bPadding; $width = $right - $left; $height = $bottom - $top; // Crop image. // Source: https://github.com/matiasdelellis/facerecognition/blob/master/lib/BackgroundJob/Tasks/ImageProcessingTask.php#L335"SS $image = imagecreatefromjpeg($img_path); $imageCrop = imagecrop($image, [ 'x' => (int)$left, 'y' => (int)$top, 'width' => (int)$width, 'height' => (int)$height ]); $img_crop_path = "cropped_" . $percent . "_". $img_path; imagejpeg($imageCrop, $img_crop_path, 100); // Get landmarks amd compute descriptors // Source: https://github.com/matiasdelellis/facerecognition/blob/master/lib/BackgroundJob/Tasks/ImageProcessingTask.php#L323 $landmarks = $fld->detect($img_crop_path, array( "left" => (int) $lPadding, "top" => (int) $tPadding, "right" => (int) ($width - $rPadding), "bottom" => (int) ($height - $bPadding) )); $descriptor = $fr->computeDescriptor($img_crop_path, $landmarks); // Draw rect to graphically check the margins. $green = imagecolorallocate($imageCrop, 132, 135, 28); imagerectangle($imageCrop, $lPadding, $tPadding, ($width - $rPadding), ($height - $bPadding), $green); imagejpeg($imageCrop, $img_crop_path, 100); // Calculate distance. // Source: https://github.com/matiasdelellis/facerecognition/blob/master/lib/Helper/Euclidean.php#L32 $n = count($descriptor); $sum = 0; for ($i = 0; $i < $n; $i++) { $sum += ($descriptorAligned[$i] - $descriptor[$i]) * ($descriptorAligned[$i] - $descriptor[$i]); } print($left . "; " . $top . "; " . $right . "; " . $bottom . "; " . $percent . "; " . sqrt($sum) . "\n"); } } ```

As you can see in the following image, the difference adding margin is noticeably reduced, however, I never get a zero difference against the original image.. :confused:

Although the same descriptor is not obtained, I think we should apply some margin since it improves the difference. What you think?

I attach the test, the resulting images and a cvs with the results.

crop-tests.zip

[stalker314314]

I'm back:)

I see what you are saying - we already have code for adding margins, but we give dlib already cropped image of face, and maybe we should just give it whole original image? That being said, I don't think your code (where you calculate descriptorAligned) is correct, as we first crop image to face. I cropped image as I thought it will be heavy operation[1], but if we wan't to do exactly as dlib example does, then we should not crop image when we provide it to calculate face description? And get rid of this face "acrobatics" I was doing.

I think this is proper solution, I would prefer to do whatever dlib does in examples, and so we don't have to do any padding on our (PHP) side.

[1] Premature optimization is the root of all evil:)

[matiasdelellis]

Hi @stalker314314 Also busy.. :sweat_smile:

[1] Premature optimization is the root of all evil:)

I suppose that is your "header phrase"/motto.. :wink:

Well, from the evaluations we did, we can accept that the difficult task is to detect the faces in image, ..but the detection of the landmakrs and the descriptor is relatively fast.

Do you agree to stop cutting? Maybe now do pull request and try with my server photos..

p.s.: I also started trying this, because I'm worried about so many writings on the disk. Remember that I had to replace my main disk --An SSD m2 where I have the main system to improve the speed-- and I do not want to blame it on this application. :sweat_smile:

[stalker314314]

Do you agree to stop cutting?

Yap, I was just waiting for you to confirm this is a way to go. Expect some code review soon!