estebanuri
commented
3 years ago I didn't do any face alignments in the android project. I've been testing different face alignment approaches on python scripts, and according to my tests it does improve the recognition accuracy.
here is the python code that could be ported to Android (the code is inspired on the PyImageSearch site)
For dlib’s 68-point facial landmark detector:
FACIAL_LANDMARKS_68_IDXS = OrderedDict([ ("mouth", (48, 68)), ("inner_mouth", (60, 68)), ("right_eyebrow", (17, 22)), ("left_eyebrow", (22, 27)), ("right_eye", (36, 42)), ("left_eye", (42, 48)), ("nose", (27, 36)), ("jaw", (0, 17)) ])
For dlib’s 5-point facial landmark detector:
FACIAL_LANDMARKS_5_IDXS = OrderedDict([ ("right_eye", (2, 3)), ("left_eye", (0, 1)), ("nose", 4) ])
def shape_to_np(shape, dtype="int"):
initialize the list of (x, y)-coordinates
coords = np.zeros((shape.num_parts, 2), dtype=dtype)
# loop over all facial landmarks and convert them
# to a 2-tuple of (x, y)-coordinates
for i in range(0, shape.num_parts):
coords[i] = (shape.part(i).x, shape.part(i).y)
# return the list of (x, y)-coordinates
return coordsdef _align_face_dolap(models, image, bbox, output_size=(160, 160), eye_x=0.3, eye_y=0.35):
output_size = image.shape[1],image.shape[0]
shape_predictor = models
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
x0, y0, x1, y1 = bbox
rect = dlib.rectangle(left=int(x0), top=int(y0), right=int(x1),bottom=int(y1)) face_shape = shape_predictor(gray, rect) face_shape = shape_to_np(face_shape)
if len(face_shape) == 68:
# extract the left and right eye (x, y)-coordinates
(l_start, l_end) = FACIAL_LANDMARKS_68_IDXS["left_eye"]
(r_start, r_end) = FACIAL_LANDMARKS_68_IDXS["right_eye"]
(n_start, n_end) = FACIAL_LANDMARKS_68_IDXS["nose"]
else:
(l_start, l_end) = FACIAL_LANDMARKS_5_IDXS["left_eye"]
(r_start, r_end) = FACIAL_LANDMARKS_5_IDXS["right_eye"]
nose_idx = FACIAL_LANDMARKS_5_IDXS["nose"]
n_start, n_end = (nose_idx, nose_idx + 1)
left_eye_pts = face_shape[l_start:l_end]
right_eye_pts = face_shape[r_start:r_end]
#nose_pts = face_shape[n_start:n_end]
# compute the center of mass for each eye
#left_eye_center = left_eye_pts.mean(axis=0).astype("int")
#right_eye_center = right_eye_pts.mean(axis=0).astype("int")
left_eye_center = left_eye_pts.mean(axis=0)
right_eye_center = right_eye_pts.mean(axis=0)
#nose_center = nose_pts.mean(axis=0).astype("int")
# compute the angle between the eye centroids
dY = right_eye_center[1] - left_eye_center[1]
dX = right_eye_center[0] - left_eye_center[0]
angle = np.degrees(np.arctan2(dY, dX)) - 180
#angle = 0
# compute the desired right eye x-coordinate based on the
# desired x-coordinate of the left eye
#left_eye_x = 0.35
# Warning!, since the face is facing front the left eye is on theright side of the image
right_eye_x = eye_x
left_eye_x = 1.0 - right_eye_x
out_w, out_h = output_size
# determine the scale of the new resulting image by taking
# the ratio of the distance between eyes in the *current*
# image to the ratio of distance between eyes in the
# *desired* image
dist = np.sqrt((dX ** 2) + (dY ** 2))
desired_dist = (left_eye_x - right_eye_x)
desired_dist *= out_w
#scale = 1
scale = desired_dist / dist
# scale = output_size / (2 * np.linalg.norm(nose_center))
# compute center (x, y)-coordinates (i.e., the median point)
# between the two eyes in the input image
eyes_center = ((left_eye_center[0] + right_eye_center[0]) / 2,
(left_eye_center[1] + right_eye_center[1]) / 2)
# # grab the rotation matrix for rotating and scaling the face
#
H = cv2.getRotationMatrix2D(eyes_center, angle, scale)
#
# # update the translation component of the matrix
tX = out_w * 0.5
#tY = out_h * eyes_center[1]
tY = out_h * eye_y
# H[0, 2] += (tX - eyes_center[0])
# H[1, 2] += (tY - eyes_center[1])
H[0, 2] += tX - eyes_center[0]
H[1, 2] += tY - eyes_center[1]
ret = cv2.warpAffine(image, H, output_size)
# return the aligned face
return retOn Mon, Sep 21, 2020 at 2:17 AM ybloch notifications@github.com wrote:
Thank you for the amazing project! I did not see that you did face alignment based on landmarks before feeding the model, did I miss it in the code? Are there any plans to add this?
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/estebanuri/face_recognition/issues/13, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACFVH2LZHBJ5XMYK7AB6LYTSG3OVBANCNFSM4RUDB2UQ .
ybloch
raja259
Thank you for the amazing project! I did not see that you did face alignment based on landmarks before feeding the model, did I miss it in the code? Are there any plans to add this?