search

google-ai-edge / mediapipe

Cross-platform, customizable ML solutions for live and streaming media.
https://ai.google.dev/edge/mediapipe
Apache License 2.0
27.78k stars 5.18k forks source link

hand_tracking_gpu build successfully but can't run #5593

Open qinweisen opened 3 months ago

qinweisen commented 3 months ago

OS Platform and Distribution

Linux Ubuntu 24.04

Compiler version

gcc version 13.2.0 (Ubuntu 13.2.0-23ubuntu4)

Programming Language and version

C++ 17

Installed using virtualenv? pip? Conda?(if python)

No response

MediaPipe version

0.10.15

Bazel version

6.1.1

XCode and Tulsi versions (if iOS)

No response

Android SDK and NDK versions (if android)

No response

Android AAR (if android)

None

OpenCV version (if running on desktop)

4.6.0

Describe the problem

I successfully build hand_tracking_gpu, but protobuf threw a fatal error during runtime.

I built hand_tracking_gpu with the following command:

$ bazel build -c opt --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 mediapipe/examples/desktop/hand_tracking:hand_tracking_gpu

And it returned "INFO: Build completed successfully, 3370 total actions"

But while I tried to run with the following command:

$ GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_gpu --calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_desktop_live_gpu.pbtxt --input_video_path=input.mp4 --output_video_path=output.mp4

Libprotobuf threw a fatal error, complete logs are as follows.

By the way, I'v already download ".tflite" files and put them in: "mediapipe/modules/hand_landmark" and "mediapipe/modules/palm_detection".

Complete Logs

WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1724903246.266829   26469 demo_run_graph_main_gpu.cc:54] Get calculator graph config contents: # MediaPipe graph that performs multi-hand tracking with TensorFlow Lite on GPU.
# Used in the examples in
# mediapipe/examples/android/src/java/com/mediapipe/apps/handtrackinggpu.

# GPU image. (GpuBuffer)
input_stream: "input_video"

# GPU image. (GpuBuffer)
output_stream: "output_video"
# Collection of detected/predicted hands, each represented as a list of
# landmarks. (std::vector<NormalizedLandmarkList>)
output_stream: "hand_landmarks"

# Generates side packet cotaining max number of hands to detect/track.
node {
  calculator: "ConstantSidePacketCalculator"
  output_side_packet: "PACKET:num_hands"
  node_options: {
    [type.googleapis.com/mediapipe.ConstantSidePacketCalculatorOptions]: {
      packet { int_value: 2 }
    }
  }
}

# Detects/tracks hand landmarks.
node {
  calculator: "HandLandmarkTrackingGpu"
  input_stream: "IMAGE:input_video"
  input_side_packet: "NUM_HANDS:num_hands"
  output_stream: "LANDMARKS:hand_landmarks"
  output_stream: "HANDEDNESS:handedness"
  output_stream: "PALM_DETECTIONS:palm_detections"
  output_stream: "HAND_ROIS_FROM_LANDMARKS:hand_rects_from_landmarks"
  output_stream: "HAND_ROIS_FROM_PALM_DETECTIONS:hand_rects_from_palm_detections"
}

# Subgraph that renders annotations and overlays them on top of the input
# images (see hand_renderer_gpu.pbtxt).
node {
  calculator: "HandRendererSubgraph"
  input_stream: "IMAGE:input_video"
  input_stream: "DETECTIONS:palm_detections"
  input_stream: "LANDMARKS:hand_landmarks"
  input_stream: "HANDEDNESS:handedness"
  input_stream: "NORM_RECTS:0:hand_rects_from_palm_detections"
  input_stream: "NORM_RECTS:1:hand_rects_from_landmarks"
  output_stream: "IMAGE:output_video"
}

[libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/generated_message_reflection.cc:2933] CHECK failed: file != nullptr:
terminate called after throwing an instance of 'google::protobuf::FatalException'
  what():  CHECK failed: file != nullptr:
Aborted
qinweisen commented 3 months ago

Strangely, I was able to run cpu version successfully in 0.10.14 but still can't use gpu version cause graph has errors as follow:

WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1724923248.305071    5186 demo_run_graph_main_gpu.cc:54] Get calculator graph config contents: # MediaPipe graph that performs multi-hand tracking with TensorFlow Lite on GPU.
# Used in the examples in
# mediapipe/examples/android/src/java/com/mediapipe/apps/handtrackinggpu.

# GPU image. (GpuBuffer)
input_stream: "input_video"

# GPU image. (GpuBuffer)
output_stream: "output_video"
# Collection of detected/predicted hands, each represented as a list of
# landmarks. (std::vector<NormalizedLandmarkList>)
output_stream: "hand_landmarks"

# Generates side packet cotaining max number of hands to detect/track.
node {
  calculator: "ConstantSidePacketCalculator"
  output_side_packet: "PACKET:num_hands"
  node_options: {
    [type.googleapis.com/mediapipe.ConstantSidePacketCalculatorOptions]: {
      packet { int_value: 2 }
    }
  }
}

# Detects/tracks hand landmarks.
node {
  calculator: "HandLandmarkTrackingGpu"
  input_stream: "IMAGE:input_video"
  input_side_packet: "NUM_HANDS:num_hands"
  output_stream: "LANDMARKS:hand_landmarks"
  output_stream: "HANDEDNESS:handedness"
  output_stream: "PALM_DETECTIONS:palm_detections"
  output_stream: "HAND_ROIS_FROM_LANDMARKS:hand_rects_from_landmarks"
  output_stream: "HAND_ROIS_FROM_PALM_DETECTIONS:hand_rects_from_palm_detections"
}

# Subgraph that renders annotations and overlays them on top of the input
# images (see hand_renderer_gpu.pbtxt).
node {
  calculator: "HandRendererSubgraph"
  input_stream: "IMAGE:input_video"
  input_stream: "DETECTIONS:palm_detections"
  input_stream: "LANDMARKS:hand_landmarks"
  input_stream: "HANDEDNESS:handedness"
  input_stream: "NORM_RECTS:0:hand_rects_from_palm_detections"
  input_stream: "NORM_RECTS:1:hand_rects_from_landmarks"
  output_stream: "IMAGE:output_video"
}

I0000 00:00:1724923248.305955    5186 demo_run_graph_main_gpu.cc:60] Initialize the calculator graph.
I0000 00:00:1724923248.315212    5186 demo_run_graph_main_gpu.cc:64] Initialize the GPU.
MESA: error: ZINK: failed to choose pdev
libEGL warning: egl: failed to create dri2 screen
I0000 00:00:1724923249.335691    5186 gl_context_egl.cc:85] Successfully initialized EGL. Major : 1 Minor: 5
I0000 00:00:1724923249.352707    5224 gl_context.cc:357] GL version: 3.0 (OpenGL ES 3.0 Mesa 24.0.9-0ubuntu0.1), renderer: D3D12 (Intel(R) UHD Graphics 770)
I0000 00:00:1724923249.352851    5186 demo_run_graph_main_gpu.cc:70] Initialize the camera or load the video.
I0000 00:00:1724923249.363143    5186 demo_run_graph_main_gpu.cc:91] Start running the calculator graph.
I0000 00:00:1724923249.364231    5186 demo_run_graph_main_gpu.cc:96] Start grabbing and processing frames.
INFO: Initialized TensorFlow Lite runtime.
INFO: Created TensorFlow Lite delegate for GPU.
E0000 00:00:1724923249.397386    5186 demo_run_graph_main_gpu.cc:199] Failed to run the graph: Graph has errors:
Calculator::Open() for node "handlandmarktrackinggpu__palmdetectiongpu__inferencecalculator__handlandmarktrackinggpu__palmdetectiongpu__InferenceCalculator" failed: ; RET_CHECK failure (mediapipe/calculators/tensor/inference_calculator_gl.cc:194) (TfLiteGpuDelegateBindBufferToTensor( delegate_.get(), gpu_buffers_in_.back()->GetOpenGlBufferWriteView().name(), interpreter_->inputs()[i]))==(kTfLiteOk)
qinweisen commented 2 months ago

Strangely, I was able to run cpu version successfully in 0.10.14 but still can't use gpu version cause graph has errors as follow:

WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1724923248.305071    5186 demo_run_graph_main_gpu.cc:54] Get calculator graph config contents: # MediaPipe graph that performs multi-hand tracking with TensorFlow Lite on GPU.
# Used in the examples in
# mediapipe/examples/android/src/java/com/mediapipe/apps/handtrackinggpu.

# GPU image. (GpuBuffer)
input_stream: "input_video"

# GPU image. (GpuBuffer)
output_stream: "output_video"
# Collection of detected/predicted hands, each represented as a list of
# landmarks. (std::vector<NormalizedLandmarkList>)
output_stream: "hand_landmarks"

# Generates side packet cotaining max number of hands to detect/track.
node {
  calculator: "ConstantSidePacketCalculator"
  output_side_packet: "PACKET:num_hands"
  node_options: {
    [type.googleapis.com/mediapipe.ConstantSidePacketCalculatorOptions]: {
      packet { int_value: 2 }
    }
  }
}

# Detects/tracks hand landmarks.
node {
  calculator: "HandLandmarkTrackingGpu"
  input_stream: "IMAGE:input_video"
  input_side_packet: "NUM_HANDS:num_hands"
  output_stream: "LANDMARKS:hand_landmarks"
  output_stream: "HANDEDNESS:handedness"
  output_stream: "PALM_DETECTIONS:palm_detections"
  output_stream: "HAND_ROIS_FROM_LANDMARKS:hand_rects_from_landmarks"
  output_stream: "HAND_ROIS_FROM_PALM_DETECTIONS:hand_rects_from_palm_detections"
}

# Subgraph that renders annotations and overlays them on top of the input
# images (see hand_renderer_gpu.pbtxt).
node {
  calculator: "HandRendererSubgraph"
  input_stream: "IMAGE:input_video"
  input_stream: "DETECTIONS:palm_detections"
  input_stream: "LANDMARKS:hand_landmarks"
  input_stream: "HANDEDNESS:handedness"
  input_stream: "NORM_RECTS:0:hand_rects_from_palm_detections"
  input_stream: "NORM_RECTS:1:hand_rects_from_landmarks"
  output_stream: "IMAGE:output_video"
}

I0000 00:00:1724923248.305955    5186 demo_run_graph_main_gpu.cc:60] Initialize the calculator graph.
I0000 00:00:1724923248.315212    5186 demo_run_graph_main_gpu.cc:64] Initialize the GPU.
MESA: error: ZINK: failed to choose pdev
libEGL warning: egl: failed to create dri2 screen
I0000 00:00:1724923249.335691    5186 gl_context_egl.cc:85] Successfully initialized EGL. Major : 1 Minor: 5
I0000 00:00:1724923249.352707    5224 gl_context.cc:357] GL version: 3.0 (OpenGL ES 3.0 Mesa 24.0.9-0ubuntu0.1), renderer: D3D12 (Intel(R) UHD Graphics 770)
I0000 00:00:1724923249.352851    5186 demo_run_graph_main_gpu.cc:70] Initialize the camera or load the video.
I0000 00:00:1724923249.363143    5186 demo_run_graph_main_gpu.cc:91] Start running the calculator graph.
I0000 00:00:1724923249.364231    5186 demo_run_graph_main_gpu.cc:96] Start grabbing and processing frames.
INFO: Initialized TensorFlow Lite runtime.
INFO: Created TensorFlow Lite delegate for GPU.
E0000 00:00:1724923249.397386    5186 demo_run_graph_main_gpu.cc:199] Failed to run the graph: Graph has errors:
Calculator::Open() for node "handlandmarktrackinggpu__palmdetectiongpu__inferencecalculator__handlandmarktrackinggpu__palmdetectiongpu__InferenceCalculator" failed: ; RET_CHECK failure (mediapipe/calculators/tensor/inference_calculator_gl.cc:194) (TfLiteGpuDelegateBindBufferToTensor( delegate_.get(), gpu_buffers_in_.back()->GetOpenGlBufferWriteView().name(), interpreter_->inputs()[i]))==(kTfLiteOk)

I've fixed it by changing the OpenGL default GPU device with export MESA_D3D12_DEFAULT_ADAPTER_NAME=NVIDIA, after that did ES version meet the requirements of 3.1.

qinweisen commented 2 months ago 
[libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/generated_message_reflection.cc:2933] CHECK failed: file != nullptr:
terminate called after throwing an instance of 'google::protobuf::FatalException'
  what():  CHECK failed: file != nullptr:
Aborted

I didn't do anything and the problem disappeared, but when I tried to build a dynamic link library, it reappeared. It seems that the error is thrown by the following code:

mediapipe::CalculatorGraphConfig config =
        mediapipe::ParseTextProtoOrDie<mediapipe::CalculatorGraphConfig>(calculator_graph_config_contents);

and nullptr caused by loading google/protobuf/descriptor.proto.

I really want to know what caused this problem.

google-ml-butler[bot] commented 2 months ago

Are you satisfied with the resolution of your issue? Yes No

yiyinglai commented 2 months ago

@qinweisen I enountered the same issue https://github.com/google-ai-edge/mediapipe/issues/5655#issue-2554015725, on different platform though. Have you solved the problem?