could you help me how I could retrieve the eye's ratios correctly?

[marzi9696]

Hi.I'm trying to retrieve the eye ratio.I uncommented this line of tracker.py and did it this way and it worked perfectly:

        f = np.clip((np.mean([f_pts[0,1], f_pts[1,1]]) - np.mean([f_pts[2,1], f_pts[3,1]])) / norm_distance_y, 0, None)
        #features["eye_r"] = self.eye_r.update(f, now)

        features["eye_r"] = f

I tested so many times using a live webcam but testing it on frames gotten from a mobile camera dose not work correctly.for example it used to detect a ratio of 0.15 when eyes are closed but for some of the frames it detects the ratio of 0.38 even though eyes are closed.I also saved the image with landmarks projected on it to see if it detects the landmarks correctly and it does. what do you think the problem might be? do you think I sould some sort of preprocessing or change the way I'm calculating the ratios? I also tested it with f.eye_blink and it detected the eyes as open.returning 1 instead of 0 thank you sorry for bothering you :)

[emilianavt]

Hi! You are correct in that there are issues with the eye open/closed features. I am currently doing some ugly threshold things in C# to actually work with those values for avatar animation. You can see the code here:

https://github.com/emilianavt/OpenSeeFace/blob/1e410015e6ee303c869506da2820ecd04b1f7b0b/Examples/OpenSeeVRMDriver.cs#L1065-L1126

I'm not exactly sure where the issues are and how to fix them though.

[marzi9696]

Hi.I figured why the ratios were so variant and it's because the landmarks x,y coordinates are based on the whole frame not the detected face frame so when someone is closer to the camera the ratio gets larger and vice versa so I mapped the landmarks to the detected face frame and it got so much better but still it's variant depending on the different individuals understandably. it's my code:

def lms_on_face (f,frame):
    height, width, channels = frame.shape
    lms = f.lms
    face_bbox = f.bbox
    face_x = face_bbox[0]
    face_y = face_bbox[1]
    face_w = face_bbox[2] + face_x
    face_h = face_bbox[3] + face_y
    face = frame[int(face_y):int(face_h),int(face_x):int(face_w)]
    face_h,face_w,_ = frame[int(face_y):int(face_h),int(face_x):int(face_w)].shape
    face_x = 0
    face_y = 0
    frame_bbox = (0,0,width,height)
    fx = 0
    fy = 0
    fw = width
    fh = height
    points = [36,37,38,39,40,41]

    for i , (x,y,c) in enumerate(lms):
        if i in points:
            x = translate(x,0,fw,0,face_w)
            y = translate(y,0,fh,0,face_h)
            lms[i] = (x,y,c)
    return lms

then I use the ratio formula and get an average based on both eyes.hope it helps someone.

[emilianavt]

Interesting, that makes sense! Thank you for posting your solution to this issue!

Thinking about it some more, if this was the cause, you could also try using f.pts_3d for your calculation as it should be somewhat pose invariant.

[marzi9696]

you are welcome and thank you for your suggestion but one question: I printed the shape of f.pts_3d and it was (703) instead of (683). how do I know which x,y coordinate are the eyes?

On Wed, Oct 7, 2020 at 12:40 PM Emiliana notifications@github.com wrote:

Interesting, that makes sense! Thank you for posting your solution to this issue!

Thinking about it some more, if this was the cause, you could also try using f.pts_3d for your calculation as it should be somewhat pose invariant.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/emilianavt/OpenSeeFace/issues/8#issuecomment-704802606, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQJJ5N6FEOZYNMXQJ52PWUTSJQWBHANCNFSM4SCOTCTQ .

[emilianavt]

The order of landmarks is the same. There are additional points for the guesstimated eyeball centers in pts_3d.