search

google-ai-edge / mediapipe

Cross-platform, customizable ML solutions for live and streaming media.
https://mediapipe.dev
Apache License 2.0
26.72k stars 5.08k forks source link

Blendshape scores and facial_transformation_matrixes in holistic model #5268

Open ChristianNSchmitz opened 5 months ago

ChristianNSchmitz commented 5 months ago

MediaPipe Solution (you are using)

Python version 0.10.11

Programming language

Python

Are you willing to contribute it

None

Describe the feature and the current behaviour/state

I would like to ask you whether the updated holistic model includes blend shape scores and the facial_transformation_matrixes. If not, I would like to ask if you could include these metrics. Thanks a lot! Christian

Will this change the current API? How?

No response

Who will benefit with this feature?

No response

Please specify the use cases for this feature

Research

Any Other info

No response

kuaashish commented 5 months ago

Hi @ChristianNSchmitz,

If you refer to the HolisticLandmarkerResult API documentation, you will notice that Blendshape is listed as [optional](https://developers.google.com/mediapipe/api/solutions/java/com/google/mediapipe/tasks/vision/holisticlandmarker/HolisticLandmarkerResult#faceBlendshapes()), and currently, Facial Transformation matrices are not included. However, we are actively working on updating the documentation, which will be released soon. Unfortunately, we are unable to provide an exact date at this time.

Thank you!!

ChristianNSchmitz commented 5 months ago

Dear @kuaashish, thank you very much for your response! For our use case, we would like to project the face back to a normalised space in order to make it comparable. Would you have a suggestion how to do that without the transformation matrix or to extract the transformation matrix somewhat differently?

kuaashish commented 2 months ago

Hi @ChristianNSchmitz,

I apologize for the delayed response. Could you please confirm if this issue has been resolved on your end, or if you still require assistance from us?

Thank you!!

ChristianNSchmitz commented 2 months ago

Dear @kuaashish , That is no problem at all, thanks for getting back to me! Unfortunately, due to issue #5461 we've could use the holistic model for the last months. Thus, the issue is still unresolved and we might need some more assistance (but cannot test it currently). Best Chris

yanhn commented 2 months ago

suggestion how to do that without the transformation matrix or to extract the transformation matrix somewhat differently?

Sorry to bother, I am solving a problem that might similar to yours. So would you please help? I am using the basic mediapipe model. I currenty have NormalizedLandmark_A(which could be drawn on image once mulplied by image width and image height) and facial_transformation_matrixes_A (a 4x4 ndarray). I need to put this expression onto another reference image using its facial_transformation_matrixes_B. Is there any suggestion for me to do this? Thank you.

kuaashish commented 1 month ago

Hi @ChristianNSchmitz,

Currently, Facial Transformation matrices are not part of Holistic, as it only returns Blendshape scores. Additionally, there is no straightforward method to obtain Facial Transformation matrices at this time. However, we are marking this as a feature request to keep it on our radar. Depending on future demand, we may implement these matrices.

Thank you!!