Blink Detection for Left Eye and Right Eye Seperatedly

[ahmetyasin-ozturk]

Hi Tadas, We are currently working on a project that enables user to use their eyes as mouse. We made progress on mapping features(thanks to the OpenFace, great work!) to screen coordinates with SVR.

We also want to make a shortcut for performing right-click operation as right eye blink and for left-click, left eye blink. Unfortunately we have seen that AU45 returns both eye state as closed or not.

• Is there a possible way to seperate AU45 as left eye and right eye? (Also we are running it from python, so our results may be wrong due to this!)
• What we tried so far? We tried upper eyelid and bottom eyelid coordinate difference, but in y-axis it returns really small value, so we cant overcome this issue with applying threshold ( returns 0, 1, -1 (we shifted landmarks >>4)). We thought if "eye gaze direction vector" returns 0 or null for, lets say left eye, we can say left eye is closed. But even we close our eyes, we have seen that those vectors not becomes 0 or null.

Thanks, Ahmet Yasin Öztürk

[TadasBaltrusaitis]

Apologies for the delay.

As you probably already discovered there is no way to use AUs to separate blinks for left and right eyes (this is due to lack of training data).

The things you tried are roughly the things I would have tried for such separation: Looking at the gap in eyelid landmarks (I wouldn't just look at y axis, but instead rotate the face to a canonical shape to avoid effects of scale and rotation), for example by looking at the 3D landmarks instead

The gaze will not provide the right signal here, as it is estimated based on landmarks and has no awareness of eye closing.

[ahmetyasin-ozturk]

Thanks, We have temporarily solved this issue by creating manually labeled data.

[ahujamanish]

hi @ahmetyasin-ozturk we're running into a similar issue with detecting blinks for both eyes separately. Can you share more details on how you solved the issue? Thank you so much.

[ahmetyasin-ozturk]

Hello @ahujamanish,

Sorry for late answer. Our results were not good enough for real world application since our time was limited, we couldnt really go for it. What we did is something like this:

We searched for ready to use labeled datasets, which you can reach here, there is different datasets.

We created models for both eyes by calculating average EAR [Soukupová, Tereza and Jan Cech. “Real-Time Eye Blink Detection using Facial Landmarks.” (2016)] values for each following 6 or 7 (you can play with the numbers to achieve best performance) frames.

Also we suggest to cross validating the model for optimized model.

So summarily, we fed -6 or 7- subsequential frame's avg EAR value to the SVR model and trained it with the labeled dataset .

You can obtain required eyelid positions for EAR formula with openface library already.

And lastly, i dont know how to do it correctly either but, this is probably the least correct way. If you can record videos which fits to your case, and label them, results would be much more feasible. And if you expect steady and rotation-less head among the camera, the things for you will get much more easy.