Can I detect eye blink?

[coolryanc]

Hi,

I get left and right eyes' landmarks by simply calling as below:

const detectionWithLandmarks = await faceDetectionTask.withFaceLandmarks()
const leftEye = detectionWithLandmarks.faceLandmarks.getLeftEye()
const rightEye = detectionWithLandmarks.faceLandmarks.getRightEye()

What I want is getting points from eyes then doing a simple calculation to find if a user closes or enlarge their eyes. However, I find the points from eyes don't have a large difference when I close or enlarge my eyes. Is there possible to detect detailed eye expression by this library?

[justadudewhohacks]

Hi,

Unfortunately it is not possible currently, the model doesn't approximate your eye contours correctly, when closing your eyes, since there are also not much samples in the training set for closed eyes.

[hsbnglr-zz]

Hi,

I am not able to get the lefteye or righteye landmarks. Can you please help. I used below code:

const faceDetectionTask = faceapi.detectSingleFace(videoEl, options) console.log("Left Eye landmarks===========>" + faceDetectionTask).faceLandmarks.getLeftEye());

[justadudewhohacks]

Your code is quite messed up, you should be doing something like the following:

const result = await faceapi.detectSingleFace(videoEl, options).withFaceLandmarks()
if (result) {
  console.log("Left Eye landmarks===========>" + result.landmarks.getLeftEye());
}

[hsbnglr-zz]

Thanks for quick reply. I used below code to get Mouth coordinates so that i can superimpose my image on it. I use below code to get it and it fetched me below results. Can you please confirm if _x and _y are pixcel distance from left and bottom of the webcam video. reason I am asking because i have to adjust my superimpose image based on this. Many Thanks?

Code:

---

const landmarks2 = await faceapi.detectFaceLandmarks(videoEl) console.log("Mouth position ===========>" + JSON.stringify(landmarks2.getMouth()));

Results: (Repetation of below array objects)

---

Mouth position ===========>[{"_x":334.44507598876953,"_y":283.48337173461914},{"_x":333.2390594482422,"_y":268.63128662109375},{"_x":324.09584045410156,"_y":258.9872074127197},{"_x":336.6387176513672,"_y":247.29134559631348},{"_x":337.9642868041992,"_y":241.43423080444336},{"_x":330.92708587646484,"_y":246.2387466430664},{"_x":348.2770538330078,"_y":266.6569519042969},{"_x":342.8745651245117,"_y":255.31597137451172},{"_x":350.4652786254883,"_y":278.01255226135254},{"_x":345.7722854614258,"_y":284.419641494751},{"_x":360.87818145751953,"_y":303.0000114440918},{"_x":353.9234161376953,"_y":297.5144290924072},{"_x":332.1590042114258,"_y":285.94871520996094},{"_x":337.9452896118164,"_y":249.26227569580078},{"_x":345.11146545410156,"_y":256.47751808166504},{"_x":330.61424255371094,"_y":242.48717308044434},{"_x":335.2454376220703,"_y":262.212610244751},{"_x":335.30696868896484,"_y":265.9950828552246},{"_x":340.21644592285156,"_y":273.7259101867676},{"_x":342.5278091430664,"_y":276.47120475769043}] webcam_face_tracking:178 Mouth position ===========>[{"_x":322.53273010253906,"_y":272.36589431762695},{"_x":324.67933654785156,"_y":248.39472770690918},{"_x":317.44129180908203,"_y":239.83108520507812},{"_x":331.2320327758789,"_y":224.28563117980957},{"_x":331.48548126220703,"_y":224.04098510742188},{"_x":328.16890716552734,"_y":233.9084815979004},{"_x":341.9776153564453,"_y":261.01123809814453},{"_x":333.83033752441406,"_y":258.4676170349121},{"_x":344.9219512939453,"_y":268.77044677734375},{"_x":345.3641891479492,"_y":275.1748752593994},{"_x":362.11463928222656,"_y":294.49201583862305},{"_x":355.1399612426758,"_y":288.5452651977539},{"_x":324.7887420654297,"_y":272.382173538208},{"_x":332.88780212402344,"_y":228.29031944274902},{"_x":339.87476348876953,"_y":235.70405960083008},{"_x":326.5476608276367,"_y":225.70733070373535},{"_x":328.65230560302734,"_y":256.69057846069336},{"_x":333.9023208618164,"_y":258.1000328063965},{"_x":337.1007537841797,"_y":264.8284149169922},{"_x":347.21168518066406,"_y":270.3109931945801}]

[justadudewhohacks]

Yes the coordinates are relative to the input image / frame.

[markose]

Hello. First of all thank you very much for this awesome library!!!

We are also interested in liveness recognition and I've tried some estimates based on aspect ratio calculation. Unfortunately there is still a certain rate of false negatives.

I think, that eye blink detection meight be the best solution to differentiate photos from real persons.

So the only solution seems be to import a better model for this.

Could you please describe how to train new input models or could you paste some links, where to find the information?

Kind regards, Marko

[justadudewhohacks]

The most difficult part of training a decent model is collecting the training data I would say. If you are interested in training an own model using tensorflowjs check out my article 18 Tips for Training your own Tensorflow.js Models in the Browser, which should give you a lot of useful hints on how to train your own model once you got your data set up.

[markose]

Thank you very much for the quick response!!! If I can train a good model, I'll post it here.

[markose]

I've created a quantizized model and the corresponding manifest.json.

Could you describe in short what I have to do, to load that model? Do I have to write another loader class?

I always get the following error:

expected weightMap[dense0/conv0/filters] to be a Tensor4D, instead have undefined.

Where do I have to describe the structure of the network?

The manifest looks like that:

[
    {
        "paths": [
            "blink-model-shard1"
        ],
        "weights": [
            {
                "name": "conv2d_1/kernel",
                "shape": [
                    3,
                    3,
                    1,
                    32
                ],
                "dtype": "float32",
                "quantization": {
                    "min": -0.2531706249012667,
                    "scale": 0.002163851494882621,
                    "dtype": "uint8"
                }
            },
            {
                "name": "conv2d_1/bias",
                "shape": [
                    32
                ],
                "dtype": "float32",
                "quantization": {
                    "min": -0.11578530131601819,
                    "scale": 0.0007328183627596088,
                    "dtype": "uint8"
                }
            },
            {
                "name": "conv2d_2/kernel",
                "shape": [
                    2,
                    2,
                    32,
                    64
                ],
                "dtype": "float32",
                "quantization": {
                    "min": -0.41054220059338736,
                    "scale": 0.0031824976790185066,
                    "dtype": "uint8"
                }
            },
            {
                "name": "conv2d_2/bias",
                "shape": [
                    64
                ],
                "dtype": "float32",
                "quantization": {
                    "min": -0.09900771177866878,
                    "scale": 0.0005593656032693152,
                    "dtype": "uint8"
                }
            },
            {
                "name": "conv2d_3/kernel",
                "shape": [
                    2,
                    2,
                    64,
                    128
                ],
                "dtype": "float32",
                "quantization": {
                    "min": -0.326127103730744,
                    "scale": 0.0026300572881511612,
                    "dtype": "uint8"
                }
            },
            {
                "name": "conv2d_3/bias",
                "shape": [
                    128
                ],
                "dtype": "float32",
                "quantization": {
                    "min": -0.10953535319981621,
                    "scale": 0.00045262542644552154,
                    "dtype": "uint8"
                }
            },
            {
                "name": "dense_1/kernel",
                "shape": [
                    1536,
                    512
                ],
                "dtype": "float32",
                "quantization": {
                    "min": -0.44373518789515776,
                    "scale": 0.0034398076581019983,
                    "dtype": "uint8"
                }
            },
            {
                "name": "dense_1/bias",
                "shape": [
                    512
                ],
                "dtype": "float32",
                "quantization": {
                    "min": -0.08901600627338184,
                    "scale": 0.0009890667363709094,
                    "dtype": "uint8"
                }
            },
            {
                "name": "dense_2/kernel",
                "shape": [
                    512,
                    512
                ],
                "dtype": "float32",
                "quantization": {
                    "min": -0.3647143992723203,
                    "scale": 0.004447736576491711,
                    "dtype": "uint8"
                }
            },
            {
                "name": "dense_2/bias",
                "shape": [
                    512
                ],
                "dtype": "float32",
                "quantization": {
                    "min": -0.09677081785950006,
                    "scale": 0.0010996683847670462,
                    "dtype": "uint8"
                }
            },
            {
                "name": "dense_3/kernel",
                "shape": [
                    512,
                    1
                ],
                "dtype": "float32",
                "quantization": {
                    "min": -0.2393317552173839,
                    "scale": 0.0016736486378838035,
                    "dtype": "uint8"
                }
            },
            {
                "name": "dense_3/bias",
                "shape": [
                    1
                ],
                "dtype": "float32",
                "quantization": {
                    "min": -0.03532525151968002,
                    "scale": 1.0,
                    "dtype": "uint8"
                }
            }
        ]
    }
]

[justadudewhohacks]

Which model are you trying to load? Note, that to each neural network in this repo there are corresponding model files, you can not load your own models.

[nathany]

Thanks for the effort on this. Also interested in blink detection. I ran across this article which was useful -- https://www.kairos.com/blog/how-to-use-blink-detection -- pointing out that blinks tend to happen 4+ times per minute and last 300ms. And a key being that both eyes close simultaneously.

[nathany]

Another article on blink detection: https://www.pyimagesearch.com/2017/04/24/eye-blink-detection-opencv-python-dlib/ Which utilizes dlib http://blog.dlib.net/2014/08/real-time-face-pose-estimation.html and appears to be based on the "One Millisecond Face Alignment with an Ensemble of Regression Trees" paper and the HELEN dataset http://www.ifp.illinois.edu/~vuongle2/helen/. 68 point iBUG 300-W dataset https://ibug.doc.ic.ac.uk/resources/facial-point-annotations/.

[karanahuja-android]

can i use eye aspect ratio formula to calculate for blink detection ?

[yaffaharari]

hi, I would be happy to know if I could use a blink formula too or if there are something else to ditect eyes blink

[mysteriousHerb]

Hi,

Unfortunately it is not possible currently, the model doesn't approximate your eye contours correctly, when closing your eyes, since there are also not much samples in the training set for closed eyes.

It seems like the eye contour doesn't change much at all when close the eyes. I wonder whether this is different from dlib facial landmark predictor, which uses iBUG 300-W dataset to train?

[cifaz]

I expect to be able to get eye blink data!

[karanahuja-android]

Pyimagesearch blog has python code for blink.

[yamaka]

grateful for the library, the eye blink would be interesting

[lacastorine]

detecting eye blink is the only thing stopping me from using this fantastic library :(

[sundar505]

Can we check whether eye is being opened or closed with this library. If so, can i have code sample

[nugrahari]

Hi,

Unfortunately it is not possible currently, the model doesn't approximate your eye contours correctly, when closing your eyes, since there are also not much samples in the training set for closed eyes.

your repository face-recognition.js, it's still using dlib for landmark and detect close eye correctly, but I can not acces the point of landmark, the example code its very shortly and just drawing the landmark point by line

[khalidtariq]

hi everyone i am using face-api for face detection i would like to know how can i show detected face in javascript

[khalidtariq]

@justadudewhohacks hi i am using face-api for face detection i would like to know how can i show detected face in javascript

[leozdim]

This repo helped me https://github.com/scriptma-n/eye-blink-detection-JS

[neonleo]

This repo helped me https://github.com/scriptma-n/eye-blink-detection-JS

That project is not 100% work, if you move your head with "fix" eye, the counter still count as blinking.

[dongshuai777]

@justadudewhohacks hi i am using face-api for face detection i would like to know how can i show detected face in javascript

@只是一个黑客 我正在使用face-api进行人脸检测我想知道如何在javascript中显示检测到的人脸

const videoEl = document.getElementById("myVideo"); const detections = await faceapi.detectSingleFace(videoEl, options); const faceImages = await faceapi.extractFaces(videoEl,[detections]); let faceImgUrl = '' faceImages.forEach((canvas) => { faceImgUrl = canvas.toDataURL(); }) faceImgUrl base64

[E0lee]

This repo helped me https://github.com/scriptma-n/eye-blink-detection-JS

this project check blink by detect eyes brightness change, it's not accurate. (QQ