yiyinglai
commented
1 month ago Figured out how to build it, you need to add --copt -DMEDIAPIPE_GPU_BUFFER_USE_CV_PIXEL_BUFFER so that GetCVPixelBufferRef is implemented for gpu (https://ai.google.dev/edge/mediapipe/framework/getting_started/cpp documentation needs update)
bazel build -c opt --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 --copt -DMEDIAPIPE_GPU_BUFFER_USE_CV_PIXEL_BUFFER \
mediapipe/examples/desktop/hand_tracking:hand_tracking_gpuHowever, it didn't run successfully
GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_gpu --calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_desktop_live_gpu.pbtxt(base) yiyinglai@MacBook-Air mediapipe % GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_gpu \
--calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_desktop_live_gpu.pbtxt
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1727542503.354913 8438947 demo_run_graph_main_gpu.cc:54] Get calculator graph config contents: # MediaPipe graph that performs multi-hand tracking with TensorFlow Lite on GPU.
# Used in the examples in
# mediapipe/examples/android/src/java/com/mediapipe/apps/handtrackinggpu.
# GPU image. (GpuBuffer)
input_stream: "input_video"
# GPU image. (GpuBuffer)
output_stream: "output_video"
# Collection of detected/predicted hands, each represented as a list of
# landmarks. (std::vector<NormalizedLandmarkList>)
output_stream: "hand_landmarks"
# Generates side packet cotaining max number of hands to detect/track.
node {
calculator: "ConstantSidePacketCalculator"
output_side_packet: "PACKET:num_hands"
node_options: {
[type.googleapis.com/mediapipe.ConstantSidePacketCalculatorOptions]: {
packet { int_value: 2 }
}
}
}
# Detects/tracks hand landmarks.
node {
calculator: "HandLandmarkTrackingGpu"
input_stream: "IMAGE:input_video"
input_side_packet: "NUM_HANDS:num_hands"
output_stream: "LANDMARKS:hand_landmarks"
output_stream: "HANDEDNESS:handedness"
output_stream: "PALM_DETECTIONS:palm_detections"
output_stream: "HAND_ROIS_FROM_LANDMARKS:hand_rects_from_landmarks"
output_stream: "HAND_ROIS_FROM_PALM_DETECTIONS:hand_rects_from_palm_detections"
}
# Subgraph that renders annotations and overlays them on top of the input
# images (see hand_renderer_gpu.pbtxt).
node {
calculator: "HandRendererSubgraph"
input_stream: "IMAGE:input_video"
input_stream: "DETECTIONS:palm_detections"
input_stream: "LANDMARKS:hand_landmarks"
input_stream: "HANDEDNESS:handedness"
input_stream: "NORM_RECTS:0:hand_rects_from_palm_detections"
input_stream: "NORM_RECTS:1:hand_rects_from_landmarks"
output_stream: "IMAGE:output_video"
}
I0000 00:00:1727542503.357008 8438947 demo_run_graph_main_gpu.cc:60] Initialize the calculator graph.
I0000 00:00:1727542503.363917 8438947 demo_run_graph_main_gpu.cc:64] Initialize the GPU.
I0000 00:00:1727542503.714582 8438947 gl_context.cc:357] GL version: 2.1 (2.1 Metal - 88.1), renderer: Apple M2
I0000 00:00:1727542503.714864 8438947 demo_run_graph_main_gpu.cc:70] Initialize the camera or load the video.
2024-09-29 00:55:04.730 hand_tracking_gpu[60945:8438947] WARNING: AVCaptureDeviceTypeExternal is deprecated for Continuity Cameras. Please use AVCaptureDeviceTypeContinuityCamera and add NSCameraUseContinuityCameraDeviceType to your Info.plist.
I0000 00:00:1727542507.677030 8438947 demo_run_graph_main_gpu.cc:91] Start running the calculator graph.
I0000 00:00:1727542507.677950 8438947 demo_run_graph_main_gpu.cc:96] Start grabbing and processing frames.
INFO: Created TensorFlow Lite delegate for Metal.
E0000 00:00:1727542513.340863 8439035 shader_util.cc:99] Failed to compile shader:
1 #version 330
2 #ifdef GL_ES
3 #define DEFAULT_PRECISION(p, t) precision p t;
4 #else
5 #define DEFAULT_PRECISION(p, t)
6 #define lowp
7 #define mediump
8 #define highp
9 #endif // defined(GL_ES)
10 #if __VERSION__ < 130
11 #define in attribute
12 #define out varying
13 #endif // __VERSION__ < 130
14 in vec4 position; in mediump vec4 texture_coordinate; out mediump vec2 sample_coordinate; void main() { gl_Position = position; sample_coordinate = texture_coordinate.xy; }
E0000 00:00:1727542513.340918 8439035 shader_util.cc:106] Error message: ERROR: 0:1: '' : version '330' is not supported
I0000 00:00:1727542513.341694 8438947 demo_run_graph_main_gpu.cc:188] Shutting down.
E0000 00:00:1727542513.381401 8438947 demo_run_graph_main_gpu.cc:199] Failed to run the graph: CalculatorGraph::Run() failed:
Calculator::Process() for node "handrenderersubgraph__AnnotationOverlayCalculator" failed: ; RET_CHECK failure (mediapipe/calculators/util/annotation_overlay_calculator.cc:732) program_Problem initializing the program.
OS Platform and Distribution
macOS Somona 14.6
Compiler version
clang 15.0.0
Programming Language and version
C++(demo code)
Installed using virtualenv? pip? Conda?(if python)
No response
MediaPipe version
No response
Bazel version
No response
XCode and Tulsi versions (if iOS)
No response
Android SDK and NDK versions (if android)
No response
Android AAR (if android)
None
OpenCV version (if running on desktop)
No response
Describe the problem
no member named 'GetCVPixelBufferRef' in namespace 'mediapipe' when building desktop hand tracking gpu
Complete Logs
Following this guide https://ai.google.dev/edge/mediapipe/framework/getting_started/cpp, I can run desktop hand tracking cpu with no issue, but desktop hand tracking gpu could not be compiled, logs: