search

google-ai-edge / mediapipe

Cross-platform, customizable ML solutions for live and streaming media.
https://ai.google.dev/edge/mediapipe
Apache License 2.0
27.23k stars 5.13k forks source link

Extract blendshape or landmark using flame-based mesh as input #5634

Open a1e0-heewon opened 1 month ago

a1e0-heewon commented 1 month ago

MediaPipe Solution (you are using)

No response

Programming language

Python

Are you willing to contribute it

None

Describe the feature and the current behaviour/state

I currently have a mesh created with Flame and would like to use it as input to a media pipe to extract blend shapes or landmarks. Does this feature already exist or is it possible?

Will this change the current API? How?

No response

Who will benefit with this feature?

No response

Please specify the use cases for this feature

You can reconstruct a high-quality mesh into a blendshape as input. Alternatively, you can re-extract the blendshape of the modified mesh using blendshape.

Any Other info

No response

kuaashish commented 3 weeks ago

Hi @a1e0-heewon,

Could you please provide more details about this feature? A clearer explanation would help us understand your request better.

Thank you!!

a1e0-heewon commented 3 weeks ago

I currently have the VOCA, COMA, and D3DFACS datasets. However, these datasets only provide a blendshape by supplying the next mesh with the Flame model. The landmark embeddings provided by Flame cannot be converted to blend shapes because only a portion of them can be extracted. Therefore, I want to extract landmarks or blendshapes directly using the mesh.