search

asus4 / tf-lite-unity-sample

TensorFlow Lite Samples on Unity
872 stars 255 forks source link

Face mesh projection error #380

Open stephahn0916 opened 1 month ago

stephahn0916 commented 1 month ago

Environment (please complete the following information):

Describe the bug When projecting the face mesh landmarks/detection, the position are wrongly placed.

To Reproduce Steps to reproduce the behavior:

  1. Load the face mesh scene
  2. Use front face detection model and face landmark model
  3. Start the scene
  4. The face mesh projection seems wrong.

Expected behavior The mesh should be aligned on the face.

Additional context In the face mesh sample cs script, there is a notice // TODO: projection is incorrect, which means that the problem was already identified.

asus4 commented 3 weeks ago

@stephahn0916 In the original MediaPipe repository, they use the Eigen library to get the projected points. I wanted to keep the dependency as simple as possible in this library. We can create a new repository with Eigen for the face mesh example, or rewrite Eigen with the Unity Burst.

https://github.com/google-ai-edge/mediapipe/blob/371ff2c157a33741343dcc9b3ae3da513f3b8cbb/mediapipe/tasks/cc/vision/face_geometry/libs/procrustes_solver.cc#L141-L198

stephahn0916 commented 3 weeks ago

I developed a procruste code based on Eigen and integrated it into Unity as a plugin (it was a few years ago, so I can search for it if needed). However, I am wondering how the procruste could help improve the alignment. When I look at this code:

https://github.com/asus4/tf-lite-unity-sample/blob/7d6c57993350e5cf8fc1f1da8ec1955d4a0c14f8/Assets/Samples/FaceMesh/FaceMeshSample.cs#L150

It means that the predictions are between 0 and 1 and you scale them to the real size of the image displayed. The Procrustes solver (if I am right) finds the transformation between the source (canonical face mesh?) and target (predicted face mesh?), but it doesn't take into account the face's deformation.

For my alignment problem, I solved it by increasing the face scaling factor which seemed too small for my face (increase to 2.0). And also modified few parameters as I moved to the last face landmaker model. It works smoothly ;) thank you for the time you put on this project.
Keep me posted about the procruste. I am curious to see how that could improve the precision.