search

vladmandic / human

Human: AI-powered 3D Face Detection & Rotation Tracking, Face Description & Recognition, Body Pose Tracking, 3D Hand & Finger Tracking, Iris Analysis, Age & Gender & Emotion Prediction, Gaze Tracking, Gesture Recognition
https://vladmandic.github.io/human/demo/index.html
MIT License
2.35k stars 323 forks source link

Low Performance when no face is detected #349

Closed Sau64 closed 1 year ago

Sau64 commented 1 year ago

I want to detect faces and gestures on mobile devices, so I'm using Blazeface. However, when I move away from the camera and there is no face detected, the frame rate drops from 60fps to 11fps, and this is affecting some other SVG animations since it runs on the main thread.

I tried using face detection on a worker, which solved the FPS issue on the main thread, but I found out that it doesn't work on iOS :(

Is there any way to improve the frame rate when no face is detected? For example, can some frames be skipped or something similar?

Great project, congrats.

Sau64 commented 1 year ago

update:

i kind of solve it using blazeface-front.json

vladmandic commented 1 year ago

a bit more info for you - face detection is much more expensive than face analysis, so human performs detection once and if face is found, it tries to keep it "in frame" using some of my tracking algorithms, so there is rarely any need to re-detect (it's tunable via config). but if face is not found, it needs to re-run face detect - thus its much slower. and there is no tunable to "skip n frames if not found".

regarding blazeface-back and blazeface-front, they are similar models, but tuned very differently. back works with smaller faces that could be anywhere in the frame so analysis of the frame is more expensive. front assumes face is actually in or near the center of the frame and larger, so it only has to try to find it there.

regarding webworkers and safari, not much i can do since apple still (its 2023???) does not fully support offscreenCanvas, so there is no way to do frame manipulation in web workers (chrome has it since version 69 in 2018).

i'm not sure what i could do to help further?

Sau64 commented 1 year ago

Oh I see, thank you for the explanation, i guess you can close this since there isn’t much it can be done.

With blazeface-front and a lower res it is going much better in mobile.

From: Vladimir Mandic @.> Sent: Wednesday, March 15, 2023 11:18 PM To: vladmandic/human @.> Cc: Sau64 @.>; Author @.> Subject: Re: [vladmandic/human] Low Performance when no face is detected (Issue #349)

a bit more info for you - face detection is much more expensive than face analysis, so human performs detection once and if face is found, it tries to keep it "in frame" using some of my tracking algorithms, so there is rarely any need to re-detect (it's tunable via config). but if face is not found, it needs to re-run face detect - thus its much slower. and there is no tunable to "skip n frames if not found".

regarding blazeface-back and blazeface-front, they are similar models, but tuned very differently. back works with smaller faces that could be anywhere in the frame so analysis of the frame is more expensive. front assumes face is actually in or near the center of the frame and larger, so it only has to try to find it there.

i'm not sure what i could do to help further?

— Reply to this email directly, view it on GitHubhttps://github.com/vladmandic/human/issues/349#issuecomment-1471179613, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACYGPUXOXDD2DJLBKCZ6OW3W4J2AJANCNFSM6AAAAAAV4QHW3I. You are receiving this because you authored the thread.Message ID: @.***>