Add blink detection

[gavinhenderson]

As part of the process of replacing EyeCommander we need to add blink detection to FaceCommander.

The code does currently have some ability to detect 'winks' but its not enabled in the UI.

We need to allow users to select 'blink' as an action similar to all the other actions

Ref:

• https://github.com/google/project-gameface/issues/17
• https://github.com/google/project-gameface/issues/23
• https://github.com/google/project-gameface/issues/34

[sjjhsjjh]

Hi @gavinhenderson , Where did the dropdown images come from please, and are there some for the blink gestures? These are the images I mean.

[gavinhenderson]

That is a good question, they look like some original google drawings to me so probably not something we have easy access too the exact same style.

~Perhaps to go with the theme of the rest of the app we can use Material UI icons?~ So looks like material ui doesnt have good icons for this.

Perhaps lets lift them from the Noun Project, something like this: https://thenounproject.com/icon/closed-eyes-1911660/ ?

You'll need to put an attribution in somewhere but i find the noun project is good for these sorts of things.

[sjjhsjjh]

I have a testable version pushed to the branch. From my own testing I can say this.

• It isn't doing a great job of recognising blink gestures if I'm wearing my spectacles.
• I find it difficult to blink with only one eye with clarity.

Happy to get feedback from anybody with a minute to try it out.

[gavinhenderson]

@sjjhsjjh Thanks for this brilliant work!

It isn't doing a great job of recognising blink gestures if I'm wearing my spectacles.

Yeah glasses can be an issue, they don't cause much issue for me when I use it, but I have fairly wide and narrow frame. That said if I sit at the wrong angle the frame can end up hiding my eye to the camera. We had a very similar issue with EyeCommander. We ended up solving it by mounting a webcam on a flexible arm that was carefully mounted at the perfect position to always see the users eyes, not ideal but it did the job.

I find it difficult to blink with only one eye with clarity.

Yeah me too. In EyeCommander we offered single eye blink and double eye blink. The vast majority of users landed on using the double eye blink. I also suspect a double eye blink would maybe be more accurate as it has more data to go on.

[sjjhsjjh]

You're welcome Gavin; happy to contribute. From what I can see in the list of gestures, the model doesn't recognise double-eye blink as such. I'm looking here https://github.com/AceCentre/FaceCommander/blob/main/src/shape_list.py#L16 That mightn't matter though. Suppose the user selects the left eye blink gesture to do an action. I expect the action would still be triggered if they blink both eyes. That could be tested I guess.

[gavinhenderson]

Hey Jim. We used the same model behind the scenes in EyeCommander. As you say, natively the model doesn't output a 'blend shape' that covers both eyes. However, you could sum the blend shape for both eyes then / 2 to get an average. Although could be worth testing as i don't how well that would actually work. This seems like the easiest option I think.

When we build EyeCommander we didn't actually use the blend shape result that MediaPipe. We opted to use the 'Landmarks' that it gives you. It gives you a landmark (x, y coord) for the upper eyelid and lower eyelid then we used that to detect a 'blink'. We used the JS wrapper for Media pipe but the same work seems possible inside of the python wrapper.

Media pipe docs on landmarks

Here is the code that EyeCommander has to 'detect a blink'. The lines of interest are L41 - L100, the other lines are all specific to the react implementation. It works by getting the landmarks for the 4 corners of each eye. It then normalises them for the distance from the camera. It then calculates a ratio of how 'closed' the eyes are based on the coords we get. You could then just do a trigger if that ratio hits a certain 'closed' threshold. However, we found that making that keeping a history of the coords and only triggering when the eye was over the 'closed' threshold for a set time was more accurate and filtered out involuntary blinks better.

The Python Media pipe gives you access to the same landmarks so in theory you could reimplement the same logic inside of FaceCommander.

[sjjhsjjh]

Hi, I had a quick test of shape-based blink detection and it works OK. Could we release that to a beta test do you think? The switch to landmark-based detection could be done later, as an improvement.

[gavinhenderson]

Could we release that to a beta test do you think

Yeah definitely, more than happy to stick with the easiest option that works!

[sjjhsjjh]

So, I guess there should be an About page in the app. It's needed for this feature to include attribution links for the eye graphics.

[gavinhenderson]

Yeah and it should also probably like to the Ace Centre website contact page and when the docs are written we can add a link in the about page too.

Here is what we had in EyeCommander

[sjjhsjjh]

OK, I have this. Links are underlined. When the mouse is hovering over a link, the address pops up at the bottom of the window. All implemented in a tkinter Text widget, and a Label for the address hint.

[sjjhsjjh]

Late additions.

• Blue links.
• Cursor changed to hand.