search

google-ai-edge / mediapipe

Cross-platform, customizable ML solutions for live and streaming media.
https://ai.google.dev/edge/mediapipe
Apache License 2.0
27.4k stars 5.15k forks source link

ARKit 52 blendshapes support request #3421

Closed GeorgeS2019 closed 1 year ago

GeorgeS2019 commented 2 years ago

Please make sure that this is a feature request.

Describe the feature and the current behavior/state:

Is there a plan to output the facial blendshapes that are compatible with the ARKit 52 blendshapes?

Will this change the current api? How?

This is an addition to the existing API.

Who will benefit with this feature?

This provides facial mocap for avatars that use the ARKit 52 blendshapes.

Please specify the use cases for this feature:

Currently, Users who are using industry standards e.g. Character Creator 3 limit themselves only facial mocap apps from Apple App Store

This feature request will democratize Facial Mocap apps to Android and make them available through Google Store.

Related discussion

Related community effort to address this unmet need for android phone users

GeorgeS2019 commented 2 years ago

Relevant attempt by VRM communities to adopt the ARKit 52 blendshapes

sureshdagooglecom commented 2 years ago

Hi @GeorgeS2019 , Blendshapes come from Xeno (https://drive.google.com/open?id=1f030M8gbXgJN-5-JhcPkDnK5ehIwl3grNhRXSZBXvjw&resourcekey=0-q6Z33dZKrau_ngUh0gJDeA) . The basic implementation is ready, but was waiting for a stable MediaPipe version of the app to submit. Pushing to M2 sounds good.

GeorgeS2019 commented 2 years ago

@sureshdagooglecom and @jiuqiant

Hi Suresh and Juiqiang, could one of you clarify what you mean by M2?

MediaPipe has become an industry standard for developers. I believe many would not need to wait for the app you refer to but would use the basic implementation immediately as part of the MediaPipe SDK.

Please PR it so the community could build or use the nightly built version (after PR merge) immediately and start testing it so we could iteratively feedback on the basic implementation.

GeorgeS2019 commented 2 years ago

For reference:

VSBuild discussion and source codes

MediaPipe Unity Plugin

zk2ly commented 2 years ago

@sureshdagooglecom Please support this feature request for https://github.com/google/mediapipe/issues/3421

zk2ly commented 2 years ago

@sureshdagooglecom

No access rights

huhai463127310 commented 2 years ago

@sureshdagooglecom

No access rights

+1

GeorgeS2019 commented 2 years ago

@kuaashish could you help address some of the questions here? Assuming that both @jiuqiant and @sureshdagooglecom are on a long holiday

brunodeangelis commented 2 years ago

+1 for this. It would really democratize the virtual character puppeteering market, allowing developers to build things we've not imagined yet

GeorgeS2019 commented 2 years ago

@brunodeangelis I am about to close this issue as our requests are being ignored.

lucasjinreal commented 2 years ago

I think 52 bs is not easily integrated with media pipe, but one can using a new model to regression these outputs with the input of 3d face pose.

brunodeangelis commented 2 years ago

I think 52 bs is not easily integrated with media pipe, but one can using a new model to regression these outputs with the input of 3d face pose.

Could you expand on this please? What type of model? Do you have an example?

emphaticaditya commented 2 years ago

any update on this guys? Is it in progress or dropped? Really looking forward to it

noeykan commented 2 years ago

I'm also waiting for this feature to be released so much... Please update this issue!

GeorgeS2019 commented 1 year ago

Hope to get an update soon on the support of this technology :-)

dandingol03 commented 1 year ago

Hi @GeorgeS2019 , Blendshapes come from Xeno (https://drive.google.com/open?id=1f030M8gbXgJN-5-JhcPkDnK5ehIwl3grNhRXSZBXvjw&resourcekey=0-q6Z33dZKrau_ngUh0gJDeA) . The basic implementation is ready, but was waiting for a stable MediaPipe version of the app to submit. Pushing to M2 sounds good.

the link of the google drvie is invalid

xuyixun21 commented 1 year ago

你好@GeorgeS2019, Blendshapes 来自 Xeno ( https://drive.google.com/open?id=1f030M8gbXgJN-5-JhcPkDnK5ehIwl3grNhRXSZBXvjw&resourcekey=0-q6Z33dZKrau_ngUh0gJDeA )。基本实现已准备就绪，但正在等待应用程序的稳定 MediaPipe 版本提交。推到 M2 听起来不错。

google drvie 的链接无效

+1

puhuajiang commented 1 year ago

Any update so far?

srcnalt commented 1 year ago

This feature can be revolutionary. Really wondering why it gets ignored.

umaruru commented 1 year ago

It seems like they are working on this: https://github.com/google/mediapipe/commit/ba10ae8410ce431f593c95986a230add675e5ccd (this was 3 days ago) But the models aren't available and there's no info on how to use it.

baronha commented 1 year ago

+1

GeorgeS2019 commented 1 year ago

ARFaceAnchor.BlendShapeLocation

  1. static let eyeBlinkLeft: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing closure of the eyelids over the left eye.

  2. static let eyeLookDownLeft: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing movement of the left eyelids consistent with a downward gaze.

  3. static let eyeLookInLeft: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing movement of the left eyelids consistent with a rightward gaze.

  4. static let eyeLookOutLeft: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing movement of the left eyelids consistent with a leftward gaze.

  5. static let eyeLookUpLeft: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing movement of the left eyelids consistent with an upward gaze.

  6. static let eyeSquintLeft: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing contraction of the face around the left eye.

  7. static let eyeWideLeft: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing a widening of the eyelids around the left eye.

  8. static let eyeBlinkRight: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing closure of the eyelids over the right eye.

  9. static let eyeLookDownRight: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing movement of the right eyelids consistent with a downward gaze.

  10. static let eyeLookInRight: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing movement of the right eyelids consistent with a leftward gaze.

  11. static let eyeLookOutRight: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing movement of the right eyelids consistent with a rightward gaze.

  12. static let eyeLookUpRight: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing movement of the right eyelids consistent with an upward gaze.

  13. static let eyeSquintRight: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing contraction of the face around the right eye.

  14. static let eyeWideRight: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing a widening of the eyelids around the right eye.

  15. static let jawForward: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing forward movement of the lower jaw.

  16. static let jawLeft: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing leftward movement of the lower jaw.

  17. static let jawRight: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing rightward movement of the lower jaw.

  18. static let jawOpen: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing an opening of the lower jaw.

  19. static let mouthClose: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing closure of the lips independent of jaw position.

  20. static let mouthFunnel: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing contraction of both lips into an open shape.

  21. static let mouthPucker: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing contraction and compression of both closed lips.

  22. static let mouthLeft: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing leftward movement of both lips together.

  23. static let mouthRight: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing rightward movement of both lips together.

  24. static let mouthSmileLeft: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing upward movement of the left corner of the mouth.

  25. static let mouthSmileRight: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing upward movement of the right corner of the mouth.

  26. static let mouthFrownLeft: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing downward movement of the left corner of the mouth.

  27. static let mouthFrownRight: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing downward movement of the right corner of the mouth.

  28. static let mouthDimpleLeft: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing backward movement of the left corner of the mouth.

  29. static let mouthDimpleRight: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing backward movement of the right corner of the mouth.

  30. static let mouthStretchLeft: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing leftward movement of the left corner of the mouth.

  31. static let mouthStretchRight: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing rightward movement of the left corner of the mouth.

  32. static let mouthRollLower: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing movement of the lower lip toward the inside of the mouth.

  33. static let mouthRollUpper: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing movement of the upper lip toward the inside of the mouth.

  34. static let mouthShrugLower: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing outward movement of the lower lip.

  35. static let mouthShrugUpper: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing outward movement of the upper lip.

  36. static let mouthPressLeft: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing upward compression of the lower lip on the left side.

  37. static let mouthPressRight: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing upward compression of the lower lip on the right side.

  38. static let mouthLowerDownLeft: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing downward movement of the lower lip on the left side.

  39. static let mouthLowerDownRight: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing downward movement of the lower lip on the right side.

  40. static let mouthUpperUpLeft: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing upward movement of the upper lip on the left side.

  41. static let mouthUpperUpRight: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing upward movement of the upper lip on the right side.

  42. static let browDownLeft: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing downward movement of the outer portion of the left eyebrow.

  43. static let browDownRight: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing downward movement of the outer portion of the right eyebrow.

  44. static let browInnerUp: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing upward movement of the inner portion of both eyebrows.

  45. static let browOuterUpLeft: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing upward movement of the outer portion of the left eyebrow.

  46. static let browOuterUpRight: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing upward movement of the outer portion of the right eyebrow.

  47. static let cheekPuff: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing outward movement of both cheeks.

  48. static let cheekSquintLeft: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing upward movement of the cheek around and below the left eye.

  49. static let cheekSquintRight: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing upward movement of the cheek around and below the right eye.

  50. static let noseSneerLeft: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing a raising of the left side of the nose around the nostril.

  51. static let noseSneerRight: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing a raising of the right

  52. static let tongueOut: ARFaceAnchor.BlendShapeLocation

    . The coefficient describing extension of the tongue.

GeorgeS2019 commented 1 year ago 
static constexpr std::array<absl::string_view, 52> kBlendshapeNames = {
    "_neutral",
    "browDownLeft",
    "browDownRight",
    "browInnerUp",
    "browOuterUpLeft",
    "browOuterUpRight",
    "cheekPuff",
    "cheekSquintLeft",
    "cheekSquintRight",
    "eyeBlinkLeft",
    "eyeBlinkRight",
    "eyeLookDownLeft",
    "eyeLookDownRight",
    "eyeLookInLeft",
    "eyeLookInRight",
    "eyeLookOutLeft",
    "eyeLookOutRight",
    "eyeLookUpLeft",
    "eyeLookUpRight",
    "eyeSquintLeft",
    "eyeSquintRight",
    "eyeWideLeft",
    "eyeWideRight",
    "jawForward",
    "jawLeft",
    "jawOpen",
    "jawRight",
    "mouthClose",
    "mouthDimpleLeft",
    "mouthDimpleRight",
    "mouthFrownLeft",
    "mouthFrownRight",
    "mouthFunnel",
    "mouthLeft",
    "mouthLowerDownLeft",
    "mouthLowerDownRight",
    "mouthPressLeft",
    "mouthPressRight",
    "mouthPucker",
    "mouthRight",
    "mouthRollLower",
    "mouthRollUpper",
    "mouthShrugLower",
    "mouthShrugUpper",
    "mouthSmileLeft",
    "mouthSmileRight",
    "mouthStretchLeft",
    "mouthStretchRight",
    "mouthUpperUpLeft",
    "mouthUpperUpRight",
    "noseSneerLeft",
    "noseSneerRight"};
GeorgeS2019 commented 1 year ago

All 52 blendshape coefficients:

GeorgeS2019 commented 1 year ago 
classification {
  index: 0
  score: 3.0398707e-05
  label: "_neutral"
}
classification {
  index: 1
  score: 0.9075446
  label: "browDownLeft"
}
classification {
  index: 2
  score: 0.88084334
  label: "browDownRight"
}
classification {
  index: 3
  score: 8.574165e-05
  label: "browInnerUp"
}
classification {
  index: 4
  score: 0.00025488975
  label: "browOuterUpLeft"
}
classification {
  index: 5
  score: 0.0003774864
  label: "browOuterUpRight"
}
classification {
  index: 6
  score: 0.0013324458
  label: "cheekPuff"
}
classification {
  index: 7
  score: 3.3356114e-06
  label: "cheekSquintLeft"
}
classification {
  index: 8
  score: 3.486574e-06
  label: "cheekSquintRight"
}
classification {
  index: 9
  score: 0.23493542
  label: "eyeBlinkLeft"
}
classification {
  index: 10
  score: 0.25156906
  label: "eyeBlinkRight"
}
classification {
  index: 11
  score: 0.31935102
  label: "eyeLookDownLeft"
}
classification {
  index: 12
  score: 0.3578726
  label: "eyeLookDownRight"
}
classification {
  index: 13
  score: 0.040764388
  label: "eyeLookInLeft"
}
classification {
  index: 14
  score: 0.10176937
  label: "eyeLookInRight"
}
classification {
  index: 15
  score: 0.13951905
  label: "eyeLookOutLeft"
}
classification {
  index: 16
  score: 0.08599486
  label: "eyeLookOutRight"
}
classification {
  index: 17
  score: 0.04357939
  label: "eyeLookUpLeft"
}
classification {
  index: 18
  score: 0.039808873
  label: "eyeLookUpRight"
}
classification {
  index: 19
  score: 0.71037096
  label: "eyeSquintLeft"
}
classification {
  index: 20
  score: 0.5613919
  label: "eyeSquintRight"
}
classification {
  index: 21
  score: 0.0035762936
  label: "eyeWideLeft"
}
classification {
  index: 22
  score: 0.0046946923
  label: "eyeWideRight"
}
classification {
  index: 23
  score: 0.0015334645
  label: "jawForward"
}
classification {
  index: 24
  score: 0.0015781156
  label: "jawLeft"
}
classification {
  index: 25
  score: 0.1825222
  label: "jawOpen"
}
classification {
  index: 26
  score: 0.00022096233
  label: "jawRight"
}
classification {
  index: 27
  score: 0.04180384
  label: "mouthClose"
}
classification {
  index: 28
  score: 0.03450827
  label: "mouthDimpleLeft"
}
classification {
  index: 29
  score: 0.02150588
  label: "mouthDimpleRight"
}
classification {
  index: 30
  score: 0.0004071877
  label: "mouthFrownLeft"
}
classification {
  index: 31
  score: 0.00041903765
  label: "mouthFrownRight"
}
classification {
  index: 32
  score: 0.006162456
  label: "mouthFunnel"
}
classification {
  index: 33
  score: 0.0007518647
  label: "mouthLeft"
}
classification {
  index: 34
  score: 0.020578554
  label: "mouthLowerDownLeft"
}
classification {
  index: 35
  score: 0.06688861
  label: "mouthLowerDownRight"
}
classification {
  index: 36
  score: 0.21384144
  label: "mouthPressLeft"
}
classification {
  index: 37
  score: 0.12757207
  label: "mouthPressRight"
}
classification {
  index: 38
  score: 0.0037631108
  label: "mouthPucker"
}
classification {
  index: 39
  score: 0.0017571376
  label: "mouthRight"
}
classification {
  index: 40
  score: 0.017397188
  label: "mouthRollLower"
}
classification {
  index: 41
  score: 0.073125295
  label: "mouthRollUpper"
}
classification {
  index: 42
  score: 0.009962141
  label: "mouthShrugLower"
}
classification {
  index: 43
  score: 0.006196939
  label: "mouthShrugUpper"
}
classification {
  index: 44
  score: 0.9412662
  label: "mouthSmileLeft"
}
classification {
  index: 45
  score: 0.9020016
  label: "mouthSmileRight"
}
classification {
  index: 46
  score: 0.054742802
  label: "mouthStretchLeft"
}
classification {
  index: 47
  score: 0.080990806
  label: "mouthStretchRight"
}
classification {
  index: 48
  score: 0.7372297
  label: "mouthUpperUpLeft"
}
classification {
  index: 49
  score: 0.69999576
  label: "mouthUpperUpRight"
}
classification {
  index: 50
  score: 3.5180938e-07
  label: "noseSneerLeft"
}
classification {
  index: 51
  score: 8.41376e-06
  label: "noseSneerRight"
}
GeorgeS2019 commented 1 year ago

To all users here, please encourage this project to support the latest 52 blendshapes. Do generously support the new feature request with emoji and the project with stars

GeorgeS2019 commented 1 year ago

The relevant commit was on 16th Feb

Will need to wait for the next release. image

Zhangtiande commented 1 year ago

I also saw the commit about blendshape, but I don't know whether I can use it directly now

GeorgeS2019 commented 1 year ago

The example to use it, I believe is being developed, official date is sometimes in April 2023

Zhangtiande commented 1 year ago

That's great, thanks to the developers

GeorgeS2019 commented 1 year ago

Step1 :

face_blendshapes.tflite

Step2 : according to this project

fadiaburaid commented 1 year ago

@GeorgeS2019 If you want to try the model and can't wait for it to be ported to Unity you can download it from here face_blendshapes.tflite The easiest way to test it is on Python. You also can use it in Unity without converting to ONNX by using TF Lite to Unity Plugin

GeorgeS2019 commented 1 year ago

See the mediapipe blendshapes in action

msedge_beivNJt8ur

GeorgeS2019 commented 1 year ago

Python example using blendshape

GeorgeS2019 commented 1 year ago

I need your support for this issue to allow more USERS interested of testing the blendshape in Game Engine doing it in faster iteration.

GeorgeS2019 commented 1 year ago

After close to 9 months of REQUEST and with MANY HERE supporting the IDEA <======. I am closing this issue NOW!
https://www.phizmocap.dev/

msedge_GG8r4VZSI3

baronha commented 1 year ago

After close to 9 months of REQUEST and with MANY HERE supporting the IDEA <======. I am closing this issue NOW! https://www.phizmocap.dev/

Are you using mediapipe for this project? I found it used from an archived package mocap4face of facemoji

GeorgeS2019 commented 1 year ago

@baronha

https://github.com/google/mediapipe/issues/4200#issue-1640418344

GeorgeS2019 commented 1 year ago

Consider supporting this issue

GeorgeS2019 commented 1 year ago

https://github.com/google/mediapipe/issues/4210

weeeeigen commented 1 year ago

I have installed mediapipe=0.9.2.1 by pip to use this good blendshape estimation. If anyone knows how to move it, it would be helpful if you could tell me.

FishWoWater commented 1 year ago

@GeorgeS2019 @fadiaburaid fa Thank you for your efforts to make this amazing feature possible! Do you have any python / javascript example to use this tflite model? It's seems that official guide is still in progress and the python example can't run

GeorgeS2019 commented 1 year ago

To Everyone, this is PART 2: Text To Speech to Facial BlendShapes, please support!!!!!!

kuaashish commented 1 year ago

@GeorgeS2019,

Could you let us know if we can mark issue as resolved and close? However, We are following up on another issue raised #4428. Thank you!!

GeorgeS2019 commented 1 year ago

@kuaashish Thank you for your support. I need it to present global crisis

FishWoWater commented 1 year ago

@schmidt-sebastian @kuaashish Hi, researchers! Could you please give me some hint on the training details of this blendshape estimator? I have referred to your solution page but still couldn't get the point. How did you generate the ground-truth 3D blendshapes(52 coefficients) from 3D mesh reconstruction? Is it an optimization-based logic, where a rigged face's blendshapes are optimized to deform the canonical mesh towards the captured 3D mesh?

Or is it follows the framework in this paper except that body parts are removed and skinning functions are specified by the artists instead of being learnable.

image