baby on the side, face isn't found

[vlebourl]

Hey,

I started to play with your repo yesterday, it's a lot of fun! I have one problem though, the face is never found... Especially when on the side. Here the kid sleeps (trust me 😅), but log says

No face found, depreciate queue Not voting on eyes, eye queue too short.

Any idea how to improve this? Does it have anything to do with the landmarks for instance? I'm very new to this...

Cheers!

[vlebourl]

I can see here https://github.com/google/mediapipe/blob/d7ee875356012514d8d5287a360cb8ea391ad0b2/mediapipe/python/solutions/face_mesh.py#L101-L106

It's using short range algo which might not be enough for my cam as it's quite far away... Any idea how to use the full algo?

[vlebourl]

a little moment later ^^'

[calebolson123]

Hey man, awesome you're trying it out. I'll start by saying I've only really had this thing working well for newborn -> what I've experienced so far, 8 mo. old. I imagine there are variables (guessing primarily blankets) that weren't factored into my initial approach. And that it might simply just not work in some scenarios w/o updating the detection logic to be able to handle more tricky scenarios like your first screenshot. If I get around to it, I have some rough ideas for more custom "sleepy human" detection logic ML models, however what's in the repo currently is relatively out of the box (mediapipe's pose & face tracking).

To answer your first question around landmarks: I don't think there should be anything around the indices to change here, looks like you've already done some digging, so you may have seen that they are always the same. And it doesn't look like there's a supported way to change from the short range model.

Although, I would have expected it to work in the scenario of your second screenshot w/o the blanket. There are a few knobs and variables you can tweak to tailor it to work in your environment better (not well documented in the code). If I were you, I would record some footage from your camera and test using the recorded footage instead of live feed. Then you can systematically tweak & test the following:

1) The first thing I would do is minimize the bounding box as much as you can, these variables:

x = 700
y = 125
h = 1000
w = 800

Looks like you may have already stumbled into these, but for the sake of seeing it work at all, I would try setting it to be framed perfectly around your child w/ no extra crib (not realistic obviously, but I would want to see it work first in perfect scenario, then reverse engineer our way up to whole crib if possible)

2) Play w/ the thresholds for the pose & faceMesh models:

Pose(min_detection_confidence=0.7, min_tracking_confidence=0.7)
...
FaceMesh(max_num_faces=1, refine_landmarks=True, min_detection_confidence=0.8, min_tracking_confidence=0.8)

I would start w/ the scenario in your second screenshot (no blanket) and drop the Pose() confidence threshold lower and see if you can find a value where it picks up the body's frame. Then it should move toward sleepy state.

There are some other things you can try, but I'd start there. I think ultimately to be able to handle a blanket covering majority of body + face obstruction, I need to roll my own classification model. I did something similar w/ detecting "baby sucking on pacifier" in my hungry baby alarm video. Let me know if you have any success

[peterpan192]

Hey,

I started to play with your repo yesterday, it's a lot of fun! I have one problem though, the face is never found... Especially when on the side. Here the kid sleeps (trust me 😅), but log says

No face found, depreciate queue Not voting on eyes, eye queue too short.

Any idea how to improve this? Does it have anything to do with the landmarks for instance? I'm very new to this...

Cheers!

Sorry this is no solution just another question: How do you guys get this live view with the video and the awake status going? Is there a way to achieve this in thonny?

[calebolson123]

https://github.com/calebolson123/BabySleepCoach/pull/18

This code will be merged in soon, and includes changes which make the system capable of tracking sleep even with blankets/objects in the crib obstructing the view. Try it out, also here's a new video I put out describing the changes: https://youtu.be/8i0wHA_knKc

[vlebourl]

Awesome, can't wait to try it out! Using Home Assistant and my own set of trackers (mostly sound and movement) I got my own sleep tracker for the kid, but I used your circular represetation for the long term time series, it's very helpful, I love it!

[calebolson123]

Ooo I like the lines and chart hour markings. Did you continue to use d3 for your version of the chart?

Interesting, sound huh? I'd be curious to see how you're using sound, and how reliable it is

[vlebourl]

Hey, I use python and matplotlib for the plot. The sound monitor is quite basic. It's a pi with a microphone under the bed. It monitors ambient sound to adjust a threshold, then counts when the threshold is hit and sends a webhook notification to my home assistant instance. I'd be happy to share the code in private, but it's not in any state to be published... Let me know if you're interested