search

keijiro / FaceLandmarkBarracuda

MediaPipe face landmark detection model for Unity Barracuda
Apache License 2.0
116 stars 25 forks source link

The position of the facial model is very shaky #6

Closed MolianWH closed 2 years ago

MolianWH commented 2 years ago

Thanks for your help for making texture with face mesh in this issue. I have successfully implemented it. But there also some problems with effectiveness.

  1. How to make the face mesh look more closer with the real face. I tried set transform.position = new Vector3(-0.5f, -0.5f, 0); , but the position looks very shaky.
  2. face mesh size looks not change with the distance of real face and camera.
  3. How to hidden the face mesh in scene when the mediaPipe not detect it?
  4. Eyes mesh can not close.(I can use blendshape to implement it, but I want to drive mesh) Bellow is the effect of my implement. Could you give some ideas? Which code should I look at?Thanks.

https://user-images.githubusercontent.com/33748947/173726076-212c4c36-5193-4f26-ae5c-04cba5c1e48c.mp4

keijiro commented 2 years ago

The MediaPipe Face Mesh model only supports properly-cropped face image input. That's why it doesn't work and get shaky when your face is away from the camera. See the model card for details.

https://drive.google.com/file/d/1QvwWNfFoweGVjsXF3DXzcrCnz-mx-Lha/preview

Usually it needs integrating another face detector to crop input image properly. I used BlazeFace in the FaceMeshBarracuda project.

The answer above explains why 1., 2., 3. happen.

Regarding 4., the MediaPipe Face Mesh does nothing about eyes. You have to integrate another eye detection model. I used the MediaPipe Iris model in the FaceMeshBarracuda project.

MolianWH commented 2 years ago

Wow! Thanks for your quick response! That's very helpful！