search

exadel-inc / CompreFace

Leading free and open-source face recognition system
https://exadel.com/accelerator-showcase/compreface/
Apache License 2.0
5.52k stars 754 forks source link

Inconsistent detection based on face orientation #1056

Open cvocvo opened 1 year ago

cvocvo commented 1 year ago

Describe the bug

I have a 1080p Wyze Cam v3 that is pointed down at a baby bed and I've been testing to try and determine which model is best for detecting my face. At first I thought all of the models weren't working. I am testing with submitting a picture of my face to Compreface leaning over the bed to be in view of where a baby will be. I had a thought that maybe it was the orientation.

In the regular photo orientation my face is upside down, then I copied the photo, rotated it 90, 180, and 270 degrees.

In normal orientation (0 degrees) where my face is oriented upside down, it cannot detect any faces. In 90 and 180 degrees it correctly detects my face. And in 270 degrees it doesn't detect my face.

Is this behavior normal / to be expected? If so I can try to predict orientation and maybe rotate the photo before passing it to compreface?

To Reproduce

See above description. I tried originally with the default model FaceNet and it didn't detect my face in the 0 degrees orientation, so I moved on to SubCenter-ArcFace-r100 and the testing above is done with that custom build.

Expected behavior

My face to be detected in all orientations of the photo.

Screenshots

If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

Logs

Run those commands and attach result to the ticket:

docker ps

# docker ps
CONTAINER ID   IMAGE                                       COMMAND                  CREATED        STATUS        PORTS                    NAMES
d11cd0a1526c   exadel/compreface-fe:1.1.0                  "/docker-entrypoint.…"   6 hours ago    Up 6 hours    0.0.0.0:8000->80/tcp     compreface-ui
9598f49979f0   exadel/compreface-admin:1.1.0               "sh -c 'java $ADMIN_…"   6 hours ago    Up 6 hours                             compreface-admin
d220f1373b4b   exadel/compreface-api:1.1.0                 "sh -c 'java $API_JA…"   6 hours ago    Up 6 hours                             compreface-api
ec7be1d3019e   exadel/compreface-postgres-db:1.1.0         "docker-entrypoint.s…"   6 hours ago    Up 6 hours    5432/tcp                 compreface-postgres-db
e836ba1aae3a   exadel/compreface-core:1.1.0-arcface-r100   "uwsgi --ini uwsgi.i…"   6 hours ago    Up 6 hours    3000/tcp                 compreface-core
1b637fee06de   robmarkcole/deepstack-ui:latest             "streamlit run deeps…"   8 hours ago    Up 8 hours    0.0.0.0:8501->8501/tcp   deepstack-ui
5e1196338c0c   deepquestai/deepstack:latest                "/app/server/server"     8 hours ago    Up 8 hours    0.0.0.0:81->5000/tcp     deepstack
f4e5826d50c7   portainer/agent:2.17.1                      "./agent"                28 hours ago   Up 28 hours   0.0.0.0:9001->9001/tcp   portainer_agent

docker-compose logs Not sure this is relevant; if needed I can sanitize and provide.

Additional context

Thank you -- any help / guidance / ideas are helpful!

pospielov commented 1 year ago

Good question We use FaceNet in the default build and InsightFace in custom builds under the hood. And it looks like they can't work with rotated faces. I did a quick research, and it looks like this is a quite wide problem. Usually, it is solved just by rotating the original image. It doesn't make sense to do it on the CompreFace side, as in mode cases, it will increase the latency but didn't add any improvements. I can suggest rotating images on your side somehow.

cvocvo commented 1 year ago

It doesn't make sense to do it on the CompreFace side, as in mode cases, it will increase the latency but didn't add any improvements. I can suggest rotating images on your side somehow.

It would be quite nice if we could specify during image processing to test the image in multiple orientations (0, 90, 180, 270) and report back if any faces are detected. In my use case, I'll never really know which way baby is sleeping. I did some testing and it does detect a baby face but only in 1 out of 4 orientations. Being able to just test all of them would be nice. I'm pushing images over to CompreFace directly via API with Home Assistant, but I can't find any easy way to rotate images on HA's side.

pospielov commented 1 year ago

I understand your concerns, but I believe this still should be a part of a middleware, not CompreFace. E.g., there is a great solution for real-time face recognition double-take. It makes sense if this will be a part of this solution.