about facemesh landmark's value

[bugmany]

when I calculate the eye's width and height, I found that there are much difference on platform Android and platform IOS, such as:(for example left eye and surprise expression) on android: landmark: Y(23) - Y(27) = 0.3364 - left eye height landmark: X(133) - X(33) = 0.3645 - left eye width when someone show surprise expression, his eye maybe very big, so above value is right, but on IOS platform with above rule, landmark: Y(23) - Y(27) = 0.1064 - left eye height landmark: X(133) - X(33) = 0.3345 - left eye width from IOS value, I think IOS's value is not right.

I do not know why, Could you give me advice?

[eknight7]

Hi @bugmany thank you for sharing this issue. Can you please share code snippet, a log of the results on iOS/Android, a video showing this issue? Can we reproduce it repeatedly? Also, is the subtraction on iOS side incorrect or are the landmark values incorrect?

[bugmany]

thank you eknight7, my source such as below: android: LandmarkProto.NormalizedLandmarkList landmarkList = multiFaceLandmarks.get(0); double eye_left_height = landmarkList.getLandmark(145).getY() - landmarkList.getLandmark(159).getY(); double eye_left_width = landmarkList.getLandmark(133).getX() - landmarkList.getLandmark(33).getX(); iOS: const auto& multi_face_landmarks = packet.Get<std::vector<::mediapipe::NormalizedLandmarkList>>(); const auto& landmarks = multi_face_landmarks[0]; double eye_left_height = landmarks.landmark(145).y() - landmarks.landmark(159).y(); double eye_left_width = landmarks.landmark(133).x() - landmarks.landmark(33).x();

these my code fragments，thanks for your help.

[eknight7]

@bugmany how did you decide which specific landmark IDs to use for calculating the height and with dimensions of the left eye?

[bugmany]

thanks @eknight7, I'm refer to file: mediapipe/graphs/face_mesh/calculators/face_landmarks_to_render_data_calculator.cc // Left eye. 33, 7, 7, 163, 163, 144, 144, 145, 145, 153, 153, 154, 154, 155, 155, 133, 33, 246, 246, 161, 161, 160, 160, 159, 159, 158, 158, 157, 157, 173, 173, 133, so I think those points is left eye landmark's point id.

[eknight7]

@bugmany

1. Can you please measure the width and height about both the left and right eyes and on both the platforms?
2. Can you ensure you are using the same input image on both platforms for accurate comparison?

[eknight7]

Here is a useful visualization (zoom in to see landmark IDs for the face) : https://github.com/google/mediapipe/blob/master/mediapipe/modules/face_geometry/data/canonical_face_model_uv_visualization.png

[bugmany]

@eknight7,

1. Can you please measure the width and height about both the left and right eyes and on both the platforms? --I did not measure the width and height, but I've ran many times.
2. Can you ensure you are using the same input image on both platforms for accurate comparison? --I've run the program on platform Android and IOS with the same input image or the same input realtime video.

with above the test, get the same results, android's value maybe right, and IOS's value is wrong.

else, I've saw the landmark IDs as your provided, https://github.com/google/mediapipe/blob/master/mediapipe/modules/face_geometry/data/canonical_face_model_uv_visualization.png I've used landmark IDs same as it.

[kostyaby]

Hey @bugmany,

I'd like to reiterate @eknight7's first question: can you please measure the width and height about both the left and right eyes and on both the platforms?

The reason we ask you to do that is because we think your runtime is different from MediaPipe Face Mesh Android/iOS examples and that you might be accidentally feeding Y-flipped frames into the iOS graph. You previously shared width / height only for the left eye; to confirm our hypothesis, we'd like to see similar width / height for the right eye. I believe you should see that the left eye width / height on Android should correspond to the right eye width / height on iOS and vice versa. If it's true, then it'll confirm that you are accidentally feeding Y-flipped frames into the iOS graph as face landmark prediction pipelines work exactly the same on iOS / Android / any other platform.

On the .../face_geometry/data/canonical_face_model_uv_visualization.png: it only shown UV coordinate visualization; the left / right notion for UV coordinates generally has nothing to do with the left / right notion for XYZ coordinates. Referring to the canonical face 3D model is better way to establish left / right notion.