Dlib's detector: eyes detection is inaccurate

[AndreaLanfranchi]

Consider this code:

                shape = self.model["sp"](img, detection)
                left_eye = (shape.part(2).x, shape.part(2).y)
                right_eye = (shape.part(0).x, shape.part(0).y)

The coordinates are taken from part 2 and 0 of the shape for left and right eye respectively. However the documentation says the 5 landmarks model (actually implemented) returns 2 points for each eye and 1 point for the nose. As a result the code above returns one boundary point for each eye and not the center of the eye (as other detectors do)

Imo the correct code should be

                shape = self.model["sp"](img, detection)
                left_eye = ((shape.part(2).x + shape.part(3).x) // 2), (shape.part(2).y + shape.part(3).y) // 2))
                right_eye = ((shape.part(0).x + shape.part(1).x) // 2), (shape.part(0).y + shape.part(1).y) // 2))

Also I have a question : the code above swaps the left eye with the right one. Does deepface consider left and right from the perspective of the observer or from the image perspective ?

[serengil]

Left eye means eye on the left of the image. In other words, it is the right eye of the person.

[AndreaLanfranchi]

Left eye means eye on the left of the image. In other words, it is the right eye of the person.

This is a great clarification and I believe should be put in extreme evidence in all documentation. For instance if somebody wants to use an eye's position to do some math it should be aware that left and right are "mirrored"

[AndreaLanfranchi]

Personal comment ... about the mirroring left and right I would underline that the approach in deepface is exactly the opposite of what detectors do : left is left for the person in the picture, not the observer's left. Should a person have a missing eye he/she would say the missing one is his/hers left or right

[serengil]

BTW, 0 and 1 indexes are for one eye and 2 and 3 indexes are for another eye. Using 0 and 2 indexes are okay for alignment, we cannot say that is not a bug. But as you mentioned they are not centers of eyes similar to other detectors. I agree with using the centers. That is why, I set its label enhancement instead of bug.

[serengil]

About the definition of left and right, you are right. We should adopt the approach in biology instead of observers perspective.

[serengil]

Highlighted eye location of dlib detector with the current implementation

[serengil]

Closed with PR - https://github.com/serengil/deepface/pull/1138

Also created a different ticket to set left and right eyes according to the person itself's perspective instead of observer: https://github.com/serengil/deepface/issues/1139