tongueOut is missing in blendshapes output

jiangwei221 commented 1 year ago

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

No

OS Platform and Distribution

M2 Mac

MediaPipe Tasks SDK version

No response

Task name (e.g. Image classification, Gesture recognition etc.)

Face Landmarks

Programming Language and version (e.g. C++, Python, Java)

Python

Describe the actual behavior

I tried to run the sample code at: https://github.com/googlesamples/mediapipe/blob/main/examples/face_landmarker/python/%5BMediaPipe_Python_Tasks%5D_Face_Landmarker.ipynb
The `tongueOut` is missing in the blendshapes outputs, and the blend shape at index 0 is `_neutral `. Could you add more details about `_neutral `?

Describe the expected behaviour

`tongueOut` should be in the blendshapes outputs, according to the tech spec at: https://storage.googleapis.com/mediapipe-assets/Model%20Card%20Blendshape%20V2.pdf

Standalone code/steps you may have used to try to get what you need

https://github.com/googlesamples/mediapipe/blob/main/examples/face_landmarker/python/%5BMediaPipe_Python_Tasks%5D_Face_Landmarker.ipynb

Other info / Complete Logs

No response

TigerHix commented 1 year ago

Reproduced on the JavaScript runtime too. Here are the FaceLandmarkerResult landmarks:

{
    "categories": [
        {
            "index": 0,
            "score": 4.6296000277834537e-7,
            "categoryName": "_neutral",
            "displayName": ""
        },
        {
            "index": 1,
            "score": 0.00009300197416450828,
            "categoryName": "browDownLeft",
            "displayName": ""
        },
        {
            "index": 2,
            "score": 0.00015640698256902397,
            "categoryName": "browDownRight",
            "displayName": ""
        },
        {
            "index": 3,
            "score": 0.6249369382858276,
            "categoryName": "browInnerUp",
            "displayName": ""
        },
        {
            "index": 4,
            "score": 0.7325420379638672,
            "categoryName": "browOuterUpLeft",
            "displayName": ""
        },
        {
            "index": 5,
            "score": 0.588916003704071,
            "categoryName": "browOuterUpRight",
            "displayName": ""
        },
        {
            "index": 6,
            "score": 0.000046172110160114244,
            "categoryName": "cheekPuff",
            "displayName": ""
        },
        {
            "index": 7,
            "score": 1.7091801396418305e-7,
            "categoryName": "cheekSquintLeft",
            "displayName": ""
        },
        {
            "index": 8,
            "score": 2.8026576615047816e-7,
            "categoryName": "cheekSquintRight",
            "displayName": ""
        },
        {
            "index": 9,
            "score": 0.08379079401493073,
            "categoryName": "eyeBlinkLeft",
            "displayName": ""
        },
        {
            "index": 10,
            "score": 0.09008345007896423,
            "categoryName": "eyeBlinkRight",
            "displayName": ""
        },
        {
            "index": 11,
            "score": 0.04183768108487129,
            "categoryName": "eyeLookDownLeft",
            "displayName": ""
        },
        {
            "index": 12,
            "score": 0.04601563885807991,
            "categoryName": "eyeLookDownRight",
            "displayName": ""
        },
        {
            "index": 13,
            "score": 0.023890702053904533,
            "categoryName": "eyeLookInLeft",
            "displayName": ""
        },
        {
            "index": 14,
            "score": 0.1881195306777954,
            "categoryName": "eyeLookInRight",
            "displayName": ""
        },
        {
            "index": 15,
            "score": 0.20005182921886444,
            "categoryName": "eyeLookOutLeft",
            "displayName": ""
        },
        {
            "index": 16,
            "score": 0.01575462333858013,
            "categoryName": "eyeLookOutRight",
            "displayName": ""
        },
        {
            "index": 17,
            "score": 0.1794263869524002,
            "categoryName": "eyeLookUpLeft",
            "displayName": ""
        },
        {
            "index": 18,
            "score": 0.20379243791103363,
            "categoryName": "eyeLookUpRight",
            "displayName": ""
        },
        {
            "index": 19,
            "score": 0.509730339050293,
            "categoryName": "eyeSquintLeft",
            "displayName": ""
        },
        {
            "index": 20,
            "score": 0.45855653285980225,
            "categoryName": "eyeSquintRight",
            "displayName": ""
        },
        {
            "index": 21,
            "score": 0.008235435001552105,
            "categoryName": "eyeWideLeft",
            "displayName": ""
        },
        {
            "index": 22,
            "score": 0.006482802797108889,
            "categoryName": "eyeWideRight",
            "displayName": ""
        },
        {
            "index": 23,
            "score": 0.0002498253888916224,
            "categoryName": "jawForward",
            "displayName": ""
        },
        {
            "index": 24,
            "score": 0.0012172190472483635,
            "categoryName": "jawLeft",
            "displayName": ""
        },
        {
            "index": 25,
            "score": 0.06601714342832565,
            "categoryName": "jawOpen",
            "displayName": ""
        },
        {
            "index": 26,
            "score": 0.0005452344776131213,
            "categoryName": "jawRight",
            "displayName": ""
        },
        {
            "index": 27,
            "score": 0.006105016451328993,
            "categoryName": "mouthClose",
            "displayName": ""
        },
        {
            "index": 28,
            "score": 0.03788747265934944,
            "categoryName": "mouthDimpleLeft",
            "displayName": ""
        },
        {
            "index": 29,
            "score": 0.008929312229156494,
            "categoryName": "mouthDimpleRight",
            "displayName": ""
        },
        {
            "index": 30,
            "score": 0.0000010028247743321117,
            "categoryName": "mouthFrownLeft",
            "displayName": ""
        },
        {
            "index": 31,
            "score": 0.0000018083688928527408,
            "categoryName": "mouthFrownRight",
            "displayName": ""
        },
        {
            "index": 32,
            "score": 0.011962022632360458,
            "categoryName": "mouthFunnel",
            "displayName": ""
        },
        {
            "index": 33,
            "score": 0.006822296418249607,
            "categoryName": "mouthLeft",
            "displayName": ""
        },
        {
            "index": 34,
            "score": 0.014128030277788639,
            "categoryName": "mouthLowerDownLeft",
            "displayName": ""
        },
        {
            "index": 35,
            "score": 0.0300290584564209,
            "categoryName": "mouthLowerDownRight",
            "displayName": ""
        },
        {
            "index": 36,
            "score": 0.025603007525205612,
            "categoryName": "mouthPressLeft",
            "displayName": ""
        },
        {
            "index": 37,
            "score": 0.019609909504652023,
            "categoryName": "mouthPressRight",
            "displayName": ""
        },
        {
            "index": 38,
            "score": 0.15044556558132172,
            "categoryName": "mouthPucker",
            "displayName": ""
        },
        {
            "index": 39,
            "score": 0.0007236231467686594,
            "categoryName": "mouthRight",
            "displayName": ""
        },
        {
            "index": 40,
            "score": 0.008093741722404957,
            "categoryName": "mouthRollLower",
            "displayName": ""
        },
        {
            "index": 41,
            "score": 0.02538474276661873,
            "categoryName": "mouthRollUpper",
            "displayName": ""
        },
        {
            "index": 42,
            "score": 0.0005738880718126893,
            "categoryName": "mouthShrugLower",
            "displayName": ""
        },
        {
            "index": 43,
            "score": 0.0018722142558544874,
            "categoryName": "mouthShrugUpper",
            "displayName": ""
        },
        {
            "index": 44,
            "score": 0.00007578419172205031,
            "categoryName": "mouthSmileLeft",
            "displayName": ""
        },
        {
            "index": 45,
            "score": 0.00008054033241933212,
            "categoryName": "mouthSmileRight",
            "displayName": ""
        },
        {
            "index": 46,
            "score": 0.000020466661226237193,
            "categoryName": "mouthStretchLeft",
            "displayName": ""
        },
        {
            "index": 47,
            "score": 0.0007130264420993626,
            "categoryName": "mouthStretchRight",
            "displayName": ""
        },
        {
            "index": 48,
            "score": 0.00019434058049228042,
            "categoryName": "mouthUpperUpLeft",
            "displayName": ""
        },
        {
            "index": 49,
            "score": 0.00018397865642327815,
            "categoryName": "mouthUpperUpRight",
            "displayName": ""
        },
        {
            "index": 50,
            "score": 0.0000022188494313013507,
            "categoryName": "noseSneerLeft",
            "displayName": ""
        },
        {
            "index": 51,
            "score": 1.455517519843852e-7,
            "categoryName": "noseSneerRight",
            "displayName": ""
        }
    ],
    "headIndex": -1,
    "headName": ""
}

jays0606 commented 1 year ago

From the 478 facial landmarks, tongue is not included in the mesh. They probably intentionally skipped tongueOut.

TigerHix commented 1 year ago

From the 478 facial landmarks, tongue is not included in the mesh. They probably intentionally skipped tongueOut.

The model documentation here did say the blendshape is available albeit predicted by the FaceMesh model.

ayushgdev commented 1 year ago

Hello @jiangwei221 We are able to reproduce this issue at our end. Please allow us sometime to get back to you after discussing internally.

zihaomu commented 12 months ago

Any update on this issue?

c4b4d4 commented 9 months ago

Is there an interest on adding any soon the tongueOut blendshape?

google-ai-edge / mediapipe