Inaccurate arrow of gaze estimation

[vladimirmujagic]

Hi,

I did some qualitative assessment using my own data trying to test face alignment + headpose + gaze. I merged some code from this repository and some additional code from lazer eye as you reffered to be able to render gaze arrow.

Its works well however to me it seems that the arrow is inverted in comparison to iris localization.

Code used for rendering:

Used constants

    YAW_THD = 45
    SIN_LEFT_THETA = 2 * sin(pi / 4)
    SIN_UP_THETA = sin(pi / 6)

Video processing

         while capture.isOpened():
                ret, frame = capture.read()
                if not ret:
                    break

                bboxes, _ = fd.inference(frame)

                detections = []
                for landmarks in fa.get_landmarks(frame, bboxes):
                    euler_angle = hp.get_head_pose(landmarks)
                    pitch, yaw, roll = euler_angle[:, 0]

                    eye_markers = np.take(landmarks, fa.eye_bound, axis=0)

                    eye_centers = np.average(eye_markers, axis=1)

                    eye_lengths = (landmarks[[39, 93]] - landmarks[[35, 89]])[:, 0]

                    iris_left = gs.get_mesh(frame, eye_lengths[0], eye_centers[0])
                    pupil_left, _ = gs.draw_pupil(iris_left, frame, thickness=1)

                    iris_right = gs.get_mesh(frame, eye_lengths[1], eye_centers[1])
                    pupil_right, _ = gs.draw_pupil(iris_right, frame, thickness=1)

                    pupils = np.array([pupil_left, pupil_right])

                    poi = landmarks[[35, 89]], landmarks[[39, 93]], pupils, eye_centers
                    theta, pha, delta = calculate_3d_gaze(frame, poi)

                    if yaw > 30:
                        end_mean = delta[0]
                    elif yaw < -30:
                        end_mean = delta[1]
                    else:
                        end_mean = np.average(delta, axis=0)

                    if end_mean[0] < 0:
                        zeta = arctan(end_mean[1] / end_mean[0]) + pi
                    else:
                        zeta = arctan(end_mean[1] / (end_mean[0] + 1e-7))

                    if roll < 0:
                        roll += 180
                    else:
                        roll -= 180

                    real_angle = zeta + roll * pi / 180

                    R = norm(end_mean)
                    offset = R * cos(real_angle), R * sin(real_angle)

                    landmarks[[38, 92]] = landmarks[[34, 88]] = eye_centers

                    draw_sticker(frame, offset, pupils, landmarks)
                    detections.append({
                        'landmarks': [lm.tolist() for lm in landmarks],
                        'offset': offset,
                        'pupils': [p.tolist() for p in pupils],
                        'iris_left': [il.tolist() for il in iris_left],
                        'iris_right': [ir.tolist() for ir in iris_right]
                    })

                json_result[frame_id] = detections
                sink.write(frame)
                frame_id += 1

Draw arrows + landmarks

def draw_sticker(src, offset, pupils, landmarks,
                 blink_thd=0.22,
                 arrow_color=(0, 125, 255), copy=False):
    if copy:
        src = src.copy()

    left_eye_hight = landmarks[33, 1] - landmarks[40, 1]
    left_eye_width = landmarks[39, 0] - landmarks[35, 0]

    right_eye_hight = landmarks[87, 1] - landmarks[94, 1]
    right_eye_width = landmarks[93, 0] - landmarks[89, 0]

    for mark in landmarks.reshape(-1, 2).astype(int):
        cv2.circle(src, tuple(mark), radius=1,
                   color=(0, 0, 255), thickness=-1)

    if left_eye_hight / left_eye_width > blink_thd:
        cv2.arrowedLine(src, tuple(pupils[0].astype(int)),
                        tuple((offset+pupils[0]).astype(int)), arrow_color, 2)

    if right_eye_hight / right_eye_width > blink_thd:
        cv2.arrowedLine(src, tuple(pupils[1].astype(int)),
                        tuple((offset+pupils[1]).astype(int)), arrow_color, 2)

    return src

[1996scarlet]

lol, this might be a bug. It is recommended to add head pose euler angle when estimating eye information. I will check later

[vladimirmujagic]

Any progress on this issue, would lazer eye work correctly as standalone ?