search

torinmb / mediapipe-touchdesigner

GPU Accelerated MediaPipe Plugin for TouchDesigner
MIT License
637 stars 31 forks source link

facetracking - faceMask vs transformmatrix #71

Open yrostudio opened 7 months ago

yrostudio commented 7 months ago

Hi, in you're example i'm trying to match the white faceMask to the canonical_face_model connected to the transformmatrix you put on your example. rotation, ... seems to be ok. But the Z (zoom in / out with head distance from camera. When i'm closed to the camera the canonical_face_model is smaller and bigger when i'm far for the camera.

Any idea ?

Screenshot 2024-02-23 at 00 54 06

My idea is to make a "kind of snapchat filter" with a 3D head oculter that follow the head size.

Thanks

yrostudio commented 7 months ago

I'm playing with mediapipe, amazing work !! Thanks to share ! In the face_tracking_examples we can clearly see that the torus is not following the Z- axis / distance/size of the head. How could i do that ?

Thanks a lot.

Screenshot 2024-02-26 at 11 11 27 Screenshot 2024-02-26 at 11 11 21
domisjustanumber commented 7 months ago

Hey @yrostudio thanks for the kind words and glad you're having fun with the plugin!

I found the same issues you did when I was playing with the plugin and the short answer is I don't know why it happens. It feels like it's something to do with the dimensions being different. The canonical face mask is supposedly roughly real-world-size, so has a height of 20 units ~ 20 cm. The face mask that we're animating/generating in the MediaPipe tox is based on screen normalised coordinates (0-1), so has a height of something like 0.8 units.

The transformation matrix we receive from MediaPipe is supposed to map from the canonical face model to screen coordinates, but I get the feeling there's a wrong scale being used somewhere (or I've applied it incorrectly in the face_mapping tox). I'm not quite sure what it is, but if you figure it out I'd love to know and we can roll it into the next release!

yrostudio commented 7 months ago

Hi @domisjustanumber, i found this discussion : https://github.com/google/mediapipe/issues/1895 don't know if it helps.

Tracking eye size / iris ? to get a Z scale ?

I have no clue how to help :)