Open ybloch opened 3 years ago
I didn't do any face alignments in the android project. I've been testing different face alignment approaches on python scripts, and according to my tests it does improve the recognition accuracy.
here is the python code that could be ported to Android (the code is inspired on the PyImageSearch site)
FACIAL_LANDMARKS_68_IDXS = OrderedDict([ ("mouth", (48, 68)), ("inner_mouth", (60, 68)), ("right_eyebrow", (17, 22)), ("left_eyebrow", (22, 27)), ("right_eye", (36, 42)), ("left_eye", (42, 48)), ("nose", (27, 36)), ("jaw", (0, 17)) ])
FACIAL_LANDMARKS_5_IDXS = OrderedDict([ ("right_eye", (2, 3)), ("left_eye", (0, 1)), ("nose", 4) ])
def shape_to_np(shape, dtype="int"):
coords = np.zeros((shape.num_parts, 2), dtype=dtype)
# loop over all facial landmarks and convert them
# to a 2-tuple of (x, y)-coordinates
for i in range(0, shape.num_parts):
coords[i] = (shape.part(i).x, shape.part(i).y)
# return the list of (x, y)-coordinates
return coords
def _align_face_dolap(models, image, bbox, output_size=(160, 160), eye_x=0.3, eye_y=0.35):
shape_predictor = models
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
x0, y0, x1, y1 = bbox
rect = dlib.rectangle(left=int(x0), top=int(y0), right=int(x1),
bottom=int(y1)) face_shape = shape_predictor(gray, rect) face_shape = shape_to_np(face_shape)
if len(face_shape) == 68:
# extract the left and right eye (x, y)-coordinates
(l_start, l_end) = FACIAL_LANDMARKS_68_IDXS["left_eye"]
(r_start, r_end) = FACIAL_LANDMARKS_68_IDXS["right_eye"]
(n_start, n_end) = FACIAL_LANDMARKS_68_IDXS["nose"]
else:
(l_start, l_end) = FACIAL_LANDMARKS_5_IDXS["left_eye"]
(r_start, r_end) = FACIAL_LANDMARKS_5_IDXS["right_eye"]
nose_idx = FACIAL_LANDMARKS_5_IDXS["nose"]
n_start, n_end = (nose_idx, nose_idx + 1)
left_eye_pts = face_shape[l_start:l_end]
right_eye_pts = face_shape[r_start:r_end]
#nose_pts = face_shape[n_start:n_end]
# compute the center of mass for each eye
#left_eye_center = left_eye_pts.mean(axis=0).astype("int")
#right_eye_center = right_eye_pts.mean(axis=0).astype("int")
left_eye_center = left_eye_pts.mean(axis=0)
right_eye_center = right_eye_pts.mean(axis=0)
#nose_center = nose_pts.mean(axis=0).astype("int")
# compute the angle between the eye centroids
dY = right_eye_center[1] - left_eye_center[1]
dX = right_eye_center[0] - left_eye_center[0]
angle = np.degrees(np.arctan2(dY, dX)) - 180
#angle = 0
# compute the desired right eye x-coordinate based on the
# desired x-coordinate of the left eye
#left_eye_x = 0.35
# Warning!, since the face is facing front the left eye is on the
right side of the image
right_eye_x = eye_x
left_eye_x = 1.0 - right_eye_x
out_w, out_h = output_size
# determine the scale of the new resulting image by taking
# the ratio of the distance between eyes in the *current*
# image to the ratio of distance between eyes in the
# *desired* image
dist = np.sqrt((dX ** 2) + (dY ** 2))
desired_dist = (left_eye_x - right_eye_x)
desired_dist *= out_w
#scale = 1
scale = desired_dist / dist
# scale = output_size / (2 * np.linalg.norm(nose_center))
# compute center (x, y)-coordinates (i.e., the median point)
# between the two eyes in the input image
eyes_center = ((left_eye_center[0] + right_eye_center[0]) / 2,
(left_eye_center[1] + right_eye_center[1]) / 2)
# # grab the rotation matrix for rotating and scaling the face
#
H = cv2.getRotationMatrix2D(eyes_center, angle, scale)
#
# # update the translation component of the matrix
tX = out_w * 0.5
#tY = out_h * eyes_center[1]
tY = out_h * eye_y
# H[0, 2] += (tX - eyes_center[0])
# H[1, 2] += (tY - eyes_center[1])
H[0, 2] += tX - eyes_center[0]
H[1, 2] += tY - eyes_center[1]
ret = cv2.warpAffine(image, H, output_size)
# return the aligned face
return ret
On Mon, Sep 21, 2020 at 2:17 AM ybloch notifications@github.com wrote:
Thank you for the amazing project! I did not see that you did face alignment based on landmarks before feeding the model, did I miss it in the code? Are there any plans to add this?
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/estebanuri/face_recognition/issues/13, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACFVH2LZHBJ5XMYK7AB6LYTSG3OVBANCNFSM4RUDB2UQ .
I think the reason you did not see a significant improvement is because you did not use the same method they used in the original insight-face repo ... From tests I did in Python on the original repo, I remember it had a significant effect on accuracy.
I saw in this repo he made alignment in their same method, with mtcnn, and their 5 landmarks is: right eye, left eye, nose, left side mouth, right side mouth.
Maybe if you try to use his method in your project you will see significant improvement...
Hi ybloch, as I said, the face alignment does help to gain some accuracy.
I looked at the insight face alignment code a few months ago, and there were several methods implemented. I tried some, and they did not work properly. Following your comment, I looked at the repository again and found this alignment code https://github.com/deepinsight/insightface/blob/master/common/face_align.py, that is different from what I had originally seen. If I'm not mistaken the code warps the image to match the landmarks with established positions.
I did several tests and found better results with the code that I shared in the previous email, that is similar to the one that is used in the repo you shared with me https://github.com/syaringan357/Android-MobileFaceNet-MTCNN-FaceAntiSpoofing/blob/master/app/src/main/java/com/zwp/mobilefacenet/mtcnn/Align.java, which simply rotates the image to align the eyes and then crops the centered image.
But perhaps, I'm doing something wrong while applying the insight-face method.
On Mon, Sep 21, 2020 at 12:38 PM ybloch notifications@github.com wrote:
I think the reason you did not see a significant improvement is because you did not use the same method they used in the original insight-face repo https://github.com/deepinsight/insightface ... From tests I did in Python on the original repo, I remember it had a significant effect on accuracy.
I saw in this repo https://github.com/syaringan357/Android-MobileFaceNet-MTCNN-FaceAntiSpoofing he made alignment in their same method, with mtcnn, and their 5 landmarks is: right eye, left eye, nose, left side mouth, right side mouth.
Maybe if you try to use his method in your project you will see significant improvement...
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/estebanuri/face_recognition/issues/13#issuecomment-696197222, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACFVH2N3UU4VAZETL5LG47TSG5XNZANCNFSM4RUDB2UQ .
Hello Uri, Sorry, I did not read your first comment carefully... ;) So we conclude that it has a positive effect, do you have any plans to add face alignment to your Android project?
Hi
It's a request to add face alignment to your Android project as this will increase the accuracy and also while taking picture for the recognition we can check that both eyes both ears forehead and chin landmark should have good values so recognition will take less time and more accurate and picture captured is good enough for the recognition purpose.
Currently I am facing lot of problems as side photos are getting captured and those are not recognizable and there matching criteria is higher then .75 which we have made default.
Thanks in advance
Hello @estebanuri, can you please share with me the python script you used for testing the MobileFAceNet TF-lite model?
Thank you for the amazing project! I did not see that you did face alignment based on landmarks before feeding the model, did I miss it in the code? Are there any plans to add this?