search

google-ai-edge / mediapipe

Cross-platform, customizable ML solutions for live and streaming media.
https://ai.google.dev/edge/mediapipe
Apache License 2.0
27.83k stars 5.18k forks source link

Expose runtime metric face mesh in FaceLandmarker results #5688

Closed kentoh closed 1 month ago

kentoh commented 1 month ago

MediaPipe Solution (you are using)

Mediapipe Tasks Vision Face Landmarker

Programming language

Javascript/Typescript

Are you willing to contribute it

Yes

Describe the feature and the current behaviour/state

The FaceGeometry module currently returns both the face pose transformation and the runtime face mesh, important primitives for typical AR applications. However, with the newer Face Landmarker (Tasks Vision) only returns the former (facialTransformationMatrices) when returning the results. It would be beneficial to return the runtime metric mesh as well so that it can

Will this change the current API? How?

Yes, this will return an additional payload in the FaceLandmarkerResult class containing the XYZ/UV vertex buffer of the runtime mesh.

Who will benefit with this feature?

Anyone building web AR applications with Mediapipe FaceLandmarker where (1) having an accurate occluder that conforms closely to the face is needed or (2) precise pinning of objects based on runtime landmark positions.

Please specify the use cases for this feature

Using the dynamic face mesh as an accurate occluder. Positioning objects more precisely against the dynamic runtime landmarks in 3D metric space.

Any Other info

No response

kentoh commented 1 month ago

Closing this as a duplicate of #5689 (somehow two issues were created).