A ParseTextProto error occurs in the 'LandmarksToTensorCalculator' of the media pipe blendshapes on Unreal Engine 5.

[JhinSH]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

Yes

OS Platform and Distribution

Windows

MediaPipe Tasks SDK version

2.9.5

Task name (e.g. Image classification, Gesture recognition etc.)

Blendshapes value to UE5

Programming Language and version (e.g. C++, Python, Java)

C++, pbtxt

Describe the actual behavior

ParseTextProto error

Describe the expected behaviour

LandmarksToTendsor calculator -> InferenceCalculatorCpu

Standalone code/steps you may have used to try to get what you need

I'm trying to extract the Unreal Engine Blendshape value using wonfei(https://github.com/wongfei/ue4-mediapipe-plugin)'s media pipe. Currently, I'm experiencing the following error and haven't made progress for a few days. Please help.

that's my error message.

----------------------------------------------------------------------------------------------------
LogMediaPipe: Display: Enter WorkerThread
LogMediaPipe: Display: UmpPipeline::Run
LogMediaPipe: Display: LoadGraphConfig: mediapipe/unreal/HJ_Practice_Blendshape.pbtxt;mediapipe/unreal/HJ_Practice_Blendshape_02.pbtxt;
LogMediaPipe: Display: LoadResourceFile: mediapipe/unreal/HJ_Practice_Blendshape.pbtxt
LogMediaPipe: Display: LoadResourceFile: mediapipe/unreal/HJ_Practice_Blendshape_02.pbtxt
LogMediaPipe: Display: ParseTextProto
LogMediaPipe: Error: ; mediapipe::ParseTextProto<mediapipe::CalculatorGraphConfig>(config_str, &config)
LogMediaPipe: Display: UmpPipeline::Shutdown
LogMediaPipe: Display: UmpPipeline::Shutdown OK
LogMediaPipe: Display: Leave WorkerThread

Other info / Complete Logs

input_stream: "input_video"
output_stream: "output_video"

output_stream: "multi_face_landmarks"
output_stream: "multi_face_landmarks_presence"

output_stream: "left_eye_contour_landmarks"
output_stream: "left_iris_landmarks"
output_stream: "left_eye_rect_from_landmarks"

output_stream: "right_eye_contour_landmarks"
output_stream: "right_iris_landmarks"
output_stream: "right_eye_rect_from_landmarks"

node {
  calculator: "FlowLimiterCalculator"
  input_stream: "input_video"
  input_stream: "FINISHED:face_landmarks_with_iris"
  input_stream_info: {
    tag_index: "FINISHED"
    back_edge: true
  }
  output_stream: "throttled_input_video"
}

node {
  calculator: "ImagePropertiesCalculator"
  input_stream: "IMAGE:throttled_input_video"
  output_stream: "SIZE:input_image_size"
}

node {
  calculator: "ConstantSidePacketCalculator"
  output_side_packet: "PACKET:num_faces"
  node_options: {
    [type.googleapis.com/mediapipe.ConstantSidePacketCalculatorOptions]: {
      packet { int_value: 1 }
    }
  }
}

node {
  calculator: "FaceLandmarkFrontCpu"
  input_stream: "IMAGE:throttled_input_video"
  input_side_packet: "NUM_FACES:num_faces"
  output_stream: "LANDMARKS:multi_face_landmarks"
  output_stream: "ROIS_FROM_LANDMARKS:face_rects_from_landmarks"
  output_stream: "DETECTIONS:face_detections"
  output_stream: "ROIS_FROM_DETECTIONS:face_rects_from_detections"
}

node {
  calculator: "PacketPresenceCalculator"
  input_stream: "PACKET:multi_face_landmarks"
  output_stream: "PRESENCE:multi_face_landmarks_presence"
}

node {
  calculator: "SplitNormalizedLandmarkListVectorCalculator"
  input_stream: "multi_face_landmarks"
  output_stream: "face_landmarks"
  node_options: {
    [type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
      ranges: { begin: 0 end: 1 }
      element_only: true
    }
  }
}

node {
  calculator: "SplitNormalizedRectVectorCalculator"
  input_stream: "face_rects_from_landmarks"
  output_stream: "face_rect"
  node_options: {
    [type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
      ranges: { begin: 0 end: 1 }
      element_only: true
    }
  }
}

node {
  calculator: "SplitNormalizedLandmarkListCalculator"
  input_stream: "face_landmarks"
  output_stream: "left_eye_boundary_landmarks"
  node_options: {
    [type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
      ranges: { begin: 33 end: 34 }
      ranges: { begin: 133 end: 134 }
      combine_outputs: true
    }
  }
}

node {
  calculator: "SplitNormalizedLandmarkListCalculator"
  input_stream: "face_landmarks"
  output_stream: "right_eye_boundary_landmarks"
  node_options: {
    [type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
      ranges: { begin: 362 end: 363 }
      ranges: { begin: 263 end: 264 }
      combine_outputs: true
    }
  }
}

node {
  calculator: "IrisLandmarkLeftAndRightCpu"
  input_stream: "IMAGE:input_video"
  input_stream: "LEFT_EYE_BOUNDARY_LANDMARKS:left_eye_boundary_landmarks"
  input_stream: "RIGHT_EYE_BOUNDARY_LANDMARKS:right_eye_boundary_landmarks"
  output_stream: "LEFT_EYE_CONTOUR_LANDMARKS:left_eye_contour_landmarks"
  output_stream: "LEFT_EYE_IRIS_LANDMARKS:left_iris_landmarks"
  output_stream: "LEFT_EYE_ROI:left_eye_rect_from_landmarks"
  output_stream: "RIGHT_EYE_CONTOUR_LANDMARKS:right_eye_contour_landmarks"
  output_stream: "RIGHT_EYE_IRIS_LANDMARKS:right_iris_landmarks"
  output_stream: "RIGHT_EYE_ROI:right_eye_rect_from_landmarks"
}

node {
  calculator: "ConcatenateNormalizedLandmarkListCalculator"
  input_stream: "left_eye_contour_landmarks"
  input_stream: "right_eye_contour_landmarks"
  output_stream: "refined_eye_landmarks"
}

node {
  calculator: "UpdateFaceLandmarksCalculator"
  input_stream: "NEW_EYE_LANDMARKS:refined_eye_landmarks"
  input_stream: "FACE_LANDMARKS:face_landmarks"
  output_stream: "UPDATED_FACE_LANDMARKS:updated_face_landmarks"
}

node {
  calculator: "IrisRendererCpu"
  input_stream: "IMAGE:input_video"
  input_stream: "FACE_LANDMARKS:updated_face_landmarks"
  input_stream: "EYE_LANDMARKS_LEFT:left_eye_contour_landmarks"
  input_stream: "EYE_LANDMARKS_RIGHT:right_eye_contour_landmarks"
  input_stream: "IRIS_LANDMARKS_LEFT:left_iris_landmarks"
  input_stream: "IRIS_LANDMARKS_RIGHT:right_iris_landmarks"
  input_stream: "NORM_RECT:face_rect"
  input_stream: "LEFT_EYE_RECT:left_eye_rect_from_landmarks"
  input_stream: "RIGHT_EYE_RECT:right_eye_rect_from_landmarks"
  input_stream: "DETECTIONS:face_detections"
  output_stream: "IRIS_LANDMARKS:iris_landmarks"
  output_stream: "IMAGE:output_video"
}

node {
  calculator: "ConcatenateNormalizedLandmarkListCalculator"
  input_stream: "updated_face_landmarks"
  input_stream: "iris_landmarks"
  output_stream: "face_landmarks_with_iris"
}

node {
  calculator: "SplitNormalizedLandmarkListCalculator"
  input_stream: "face_landmarks_with_iris"
  output_stream: "norm_landmarks"
  options {
    [mediapipe.SplitVectorCalculatorOptions.ext] {
      ranges {
        begin: 0
        end: 1
      }
      ranges {
        begin: 1
        end: 2
      }
      ranges {
        begin: 4
        end: 5
      }
      ranges {
        begin: 5
        end: 6
      }
      ranges {
        begin: 6
        end: 7
      }
      ranges {
        begin: 7
        end: 8
      }
      ranges {
        begin: 8
        end: 9
      }
      ranges {
        begin: 10
        end: 11
      }
      ranges {
        begin: 13
        end: 14
      }
      ranges {
        begin: 14
        end: 15
      }
      ranges {
        begin: 17
        end: 18
      }
      ranges {
        begin: 21
        end: 22
      }
      ranges {
        begin: 33
        end: 34
      }
      ranges {
        begin: 37
        end: 38
      }
      ranges {
        begin: 39
        end: 40
      }
      ranges {
        begin: 40
        end: 41
      }
      ranges {
        begin: 46
        end: 47
      }
      ranges {
        begin: 52
        end: 53
      }
      ranges {
        begin: 53
        end: 54
      }
      ranges {
        begin: 54
        end: 55
      }
      ranges {
        begin: 55
        end: 56
      }
      ranges {
        begin: 58
        end: 59
      }
      ranges {
        begin: 61
        end: 62
      }
      ranges {
        begin: 63
        end: 64
      }
      ranges {
        begin: 65
        end: 66
      }
      ranges {
        begin: 66
        end: 67
      }
      ranges {
        begin: 67
        end: 68
      }
      ranges {
        begin: 70
        end: 71
      }
      ranges {
        begin: 78
        end: 79
      }
      ranges {
        begin: 80
        end: 81
      }
      ranges {
        begin: 81
        end: 82
      }
      ranges {
        begin: 82
        end: 83
      }
      ranges {
        begin: 84
        end: 85
      }
      ranges {
        begin: 87
        end: 88
      }
      ranges {
        begin: 88
        end: 89
      }
      ranges {
        begin: 91
        end: 92
      }
      ranges {
        begin: 93
        end: 94
      }
      ranges {
        begin: 95
        end: 96
      }
      ranges {
        begin: 103
        end: 104
      }
      ranges {
        begin: 105
        end: 106
      }
      ranges {
        begin: 107
        end: 108
      }
      ranges {
        begin: 109
        end: 110
      }
      ranges {
        begin: 127
        end: 128
      }
      ranges {
        begin: 132
        end: 133
      }
      ranges {
        begin: 133
        end: 134
      }
      ranges {
        begin: 136
        end: 137
      }
      ranges {
        begin: 144
        end: 145
      }
      ranges {
        begin: 145
        end: 146
      }
      ranges {
        begin: 146
        end: 147
      }
      ranges {
        begin: 148
        end: 149
      }
      ranges {
        begin: 149
        end: 150
      }
      ranges {
        begin: 150
        end: 151
      }
      ranges {
        begin: 152
        end: 153
      }
      ranges {
        begin: 153
        end: 154
      }
      ranges {
        begin: 154
        end: 155
      }
      ranges {
        begin: 155
        end: 156
      }
      ranges {
        begin: 157
        end: 158
      }
      ranges {
        begin: 158
        end: 159
      }
      ranges {
        begin: 159
        end: 160
      }
      ranges {
        begin: 160
        end: 161
      }
      ranges {
        begin: 161
        end: 162
      }
      ranges {
        begin: 162
        end: 163
      }
      ranges {
        begin: 163
        end: 164
      }
      ranges {
        begin: 168
        end: 169
      }
      ranges {
        begin: 172
        end: 173
      }
      ranges {
        begin: 173
        end: 174
      }
      ranges {
        begin: 176
        end: 177
      }
      ranges {
        begin: 178
        end: 179
      }
      ranges {
        begin: 181
        end: 182
      }
      ranges {
        begin: 185
        end: 186
      }
      ranges {
        begin: 191
        end: 192
      }
      ranges {
        begin: 195
        end: 196
      }
      ranges {
        begin: 197
        end: 198
      }
      ranges {
        begin: 234
        end: 235
      }
      ranges {
        begin: 246
        end: 247
      }
      ranges {
        begin: 249
        end: 250
      }
      ranges {
        begin: 251
        end: 252
      }
      ranges {
        begin: 263
        end: 264
      }
      ranges {
        begin: 267
        end: 268
      }
      ranges {
        begin: 269
        end: 270
      }
      ranges {
        begin: 270
        end: 271
      }
      ranges {
        begin: 276
        end: 277
      }
      ranges {
        begin: 282
        end: 283
      }
      ranges {
        begin: 283
        end: 284
      }
      ranges {
        begin: 284
        end: 285
      }
      ranges {
        begin: 285
        end: 286
      }
      ranges {
        begin: 288
        end: 289
      }
      ranges {
        begin: 291
        end: 292
      }
      ranges {
        begin: 293
        end: 294
      }
      ranges {
        begin: 295
        end: 296
      }
      ranges {
        begin: 296
        end: 297
      }
      ranges {
        begin: 297
        end: 298
      }
      ranges {
        begin: 300
        end: 301
      }
      ranges {
        begin: 308
        end: 309
      }
      ranges {
        begin: 310
        end: 311
      }
      ranges {
        begin: 311
        end: 312
      }
      ranges {
        begin: 312
        end: 313
      }
      ranges {
        begin: 314
        end: 315
      }
      ranges {
        begin: 317
        end: 318
      }
      ranges {
        begin: 318
        end: 319
      }
      ranges {
        begin: 321
        end: 322
      }
      ranges {
        begin: 323
        end: 324
      }
      ranges {
        begin: 324
        end: 325
      }
      ranges {
        begin: 332
        end: 333
      }
      ranges {
        begin: 334
        end: 335
      }
      ranges {
        begin: 336
        end: 337
      }
      ranges {
        begin: 338
        end: 339
      }
      ranges {
        begin: 356
        end: 357
      }
      ranges {
        begin: 361
        end: 362
      }
      ranges {
        begin: 362
        end: 363
      }
      ranges {
        begin: 365
        end: 366
      }
      ranges {
        begin: 373
        end: 374
      }
      ranges {
        begin: 374
        end: 375
      }
      ranges {
        begin: 375
        end: 376
      }
      ranges {
        begin: 377
        end: 378
      }
      ranges {
        begin: 378
        end: 379
      }
      ranges {
        begin: 379
        end: 380
      }
      ranges {
        begin: 380
        end: 381
      }
      ranges {
        begin: 381
        end: 382
      }
      ranges {
        begin: 382
        end: 383
      }
      ranges {
        begin: 384
        end: 385
      }
      ranges {
        begin: 385
        end: 386
      }
      ranges {
        begin: 386
        end: 387
      }
      ranges {
        begin: 387
        end: 388
      }
      ranges {
        begin: 388
        end: 389
      }
      ranges {
        begin: 389
        end: 390
      }
      ranges {
        begin: 390
        end: 391
      }
      ranges {
        begin: 397
        end: 398
      }
      ranges {
        begin: 398
        end: 399
      }
      ranges {
        begin: 400
        end: 401
      }
      ranges {
        begin: 402
        end: 403
      }
      ranges {
        begin: 405
        end: 406
      }
      ranges {
        begin: 409
        end: 410
      }
      ranges {
        begin: 415
        end: 416
      }
      ranges {
        begin: 454
        end: 455
      }
      ranges {
        begin: 466
        end: 467
      }
      ranges {
        begin: 468
        end: 469
      }
      ranges {
        begin: 469
        end: 470
      }
      ranges {
        begin: 470
        end: 471
      }
      ranges {
        begin: 471
        end: 472
      }
      ranges {
        begin: 472
        end: 473
      }
      ranges {
        begin: 473
        end: 474
      }
      ranges {
        begin: 474
        end: 475
      }
      ranges {
        begin: 475
        end: 476
      }
      ranges {
        begin: 476
        end: 477
      }
      ranges {
        begin: 477
        end: 478
      }
      combine_outputs: true
    }
  }
}

node {
  calculator: "LandmarksToTensorCalculator"
  input_stream: "IMAGE_SIZE:input_image_size"
  input_stream: "NORM_LANDMARKS:norm_landmarks"
  output_stream: "TENSORS:face_tensor_in"
  options {
    [mediapipe.LandmarksToTensorCalculatorOptions.ext] {
      attributes: X
      attributes: Y
      flatten: false
    }
  }
}

node {
  calculator: "InferenceCalculatorCpu"
  input_stream: "TENSORS:face_tensor_in"
  output_stream: "TENSORS:face_tensor_out"
  options {
    [mediapipe.InferenceCalculatorOptions.ext] {
      model_path: "./mediapipe/modules/face_landmark/face_blendshapes.tflite"
    }
  }
}
node {
  calculator: "SplitTensorVectorCalculator"
  input_stream: "face_tensor_out"
  output_stream: "__stream_5"
  options {
    [mediapipe.SplitVectorCalculatorOptions.ext] {
      ranges {
        begin: 0
        end: 1
      }
      combine_outputs: true
    }
  }
}
node {
  calculator: "TensorsToClassificationCalculator"
  input_stream: "TENSORS:__stream_5"
  output_stream: "CLASSIFICATIONS:face_blendshapes"
  options {
    [mediapipe.TensorsToClassificationCalculatorOptions.ext] {
      min_score_threshold: -1
      top_k: 0
      label_map {
        entries {
          id: 0
          label: "_neutral"
        }
        entries {
          id: 1
          label: "browDownLeft"
        }
        entries {
          id: 2
          label: "browDownRight"
        }
        entries {
          id: 3
          label: "browInnerUp"
        }
        entries {
          id: 4
          label: "browOuterUpLeft"
        }
        entries {
          id: 5
          label: "browOuterUpRight"
        }
        entries {
          id: 6
          label: "cheekPuff"
        }
        entries {
          id: 7
          label: "cheekSquintLeft"
        }
        entries {
          id: 8
          label: "cheekSquintRight"
        }
        entries {
          id: 9
          label: "eyeBlinkLeft"
        }
        entries {
          id: 10
          label: "eyeBlinkRight"
        }
        entries {
          id: 11
          label: "eyeLookDownLeft"
        }
        entries {
          id: 12
          label: "eyeLookDownRight"
        }
        entries {
          id: 13
          label: "eyeLookInLeft"
        }
        entries {
          id: 14
          label: "eyeLookInRight"
        }
        entries {
          id: 15
          label: "eyeLookOutLeft"
        }
        entries {
          id: 16
          label: "eyeLookOutRight"
        }
        entries {
          id: 17
          label: "eyeLookUpLeft"
        }
        entries {
          id: 18
          label: "eyeLookUpRight"
        }
        entries {
          id: 19
          label: "eyeSquintLeft"
        }
        entries {
          id: 20
          label: "eyeSquintRight"
        }
        entries {
          id: 21
          label: "eyeWideLeft"
        }
        entries {
          id: 22
          label: "eyeWideRight"
        }
        entries {
          id: 23
          label: "jawForward"
        }
        entries {
          id: 24
          label: "jawLeft"
        }
        entries {
          id: 25
          label: "jawOpen"
        }
        entries {
          id: 26
          label: "jawRight"
        }
        entries {
          id: 27
          label: "mouthClose"
        }
        entries {
          id: 28
          label: "mouthDimpleLeft"
        }
        entries {
          id: 29
          label: "mouthDimpleRight"
        }
        entries {
          id: 30
          label: "mouthFrownLeft"
        }
        entries {
          id: 31
          label: "mouthFrownRight"
        }
        entries {
          id: 32
          label: "mouthFunnel"
        }
        entries {
          id: 33
          label: "mouthLeft"
        }
        entries {
          id: 34
          label: "mouthLowerDownLeft"
        }
        entries {
          id: 35
          label: "mouthLowerDownRight"
        }
        entries {
          id: 36
          label: "mouthPressLeft"
        }
        entries {
          id: 37
          label: "mouthPressRight"
        }
        entries {
          id: 38
          label: "mouthPucker"
        }
        entries {
          id: 39
          label: "mouthRight"
        }
        entries {
          id: 40
          label: "mouthRollLower"
        }
        entries {
          id: 41
          label: "mouthRollUpper"
        }
        entries {
          id: 42
          label: "mouthShrugLower"
        }
        entries {
          id: 43
          label: "mouthShrugUpper"
        }
        entries {
          id: 44
          label: "mouthSmileLeft"
        }
        entries {
          id: 45
          label: "mouthSmileRight"
        }
        entries {
          id: 46
          label: "mouthStretchLeft"
        }
        entries {
          id: 47
          label: "mouthStretchRight"
        }
        entries {
          id: 48
          label: "mouthUpperUpLeft"
        }
        entries {
          id: 49
          label: "mouthUpperUpRight"
        }
        entries {
          id: 50
          label: "noseSneerLeft"
        }
        entries {
          id: 51
          label: "noseSneerRight"
        }
      }
    }
  }
}

it's my pbtxt.

[kuaashish]

Hi @JhinSH,

We have not tested this scenario and do not have official support for Unreal Engine. It would be best to raise the issue in the same repository and follow up there for further assistance. Unfortunately, we will not be able to assist further. Hope you can understand.

Thank you!!