GPU support for pose estimation in python

[FredrikHolsten]

Hi! Great work on mediapipe!

Python users could benefit from running the pose estimator on GPU, especially when using model complexity 2 (heavy).

I have seen lots of people trying to make this work, but it's unclear if anyone actually have made it and how to reproduce with the latest version.

https://github.com/google/mediapipe/issues/1740 https://github.com/google/mediapipe/issues/1042 https://github.com/google/mediapipe/issues/3106 https://github.com/google/mediapipe/issues/1651 https://github.com/google/mediapipe/issues/2691 https://github.com/google/mediapipe/issues/1829

Thanks again for a great framework.

[sureshdagooglecom]

Hi @FredrikHolsten , MediaPipe Python package only contains CPU graphs. If you need to run any GPU graphs, see https://github.com/google/mediapipe/issues/1651 (comment).

[FredrikHolsten]

Hi @sureshdagooglecom, issue #1651 demonstrates the problem as it is not straight forward to make it work for the pose estimator (not holistic) and it does not apply to the latest version. Are there any plans on making GPU support for python official, or make an updated guide on how to make it work?

[ChimeSkov]

I've tried following the same issue with no luck

[google-ml-butler[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

[xiong-jie-y]

I successfully run some of the examples(facemesh, face detection, pose, holistic, hand) in python on GPU in recent mediapipe :slightly_smiling_face: The code is in this branch and the procedure to install the python module so far is

git clone git@github.com:xiong-jie-y/mediapipe.git
cd mediapipe
git checkout add_gpu_support_for_python
python setup.py gen_protos && python setup.py install --link-opencv

[Programmerwyl]

@xiong-jie-y

According to your process, I compiled successfully. I also found that GPU was used, but the program ran very slowly. Do you know the reason.

hand_landmarks.py
time 0.32150912284851074 time 0.32856130599975586 time 0.32004642486572266 time 0.32836365699768066 time 0.3403322696685791 time 0.3230917453765869 time 0.49794578552246094 time 0.1529250144958496 time 0.3360939025878906 time 0.3474714756011963 time 0.35848188400268555 time 0.36338329315185547 time 0.3516552448272705 time 0.3686668872833252

for face_mesh.py

time 0.04909253120422363 time 0.14416909217834473 time 0.11721205711364746 time 0.1072845458984375 time 0.1448345184326172 time 0.12059855461120605 time 0.11280131340026855

[Programmerwyl]

nvcc -V

nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2021 NVIDIA Corporation Built on Thu_Nov_18_09:45:30_PST_2021 Cuda compilation tools, release 11.5, V11.5.119 Build cuda_11.5.r11.5/compiler.30672275_0

[Programmerwyl]

1. install opengl

   sudo apt-get install build-essential libgl1-mesa-dev
   sudo apt-get install libglew-dev libsdl2-dev libsdl2-image-dev libglm-dev libfreetype6-dev
   sudo apt-get install libglfw3-dev libglfw3
   sudo apt-get install libgl1-mesa-dev
   sudo apt-get install libglu1-mesa-dev
   sudo apt-get install freeglut3-dev
   sudo apt install mesa-utils
   sudo apt-get install libgtkglext1 libgtkglext1-dev
   glxinfo | grep OpenGL

   install opencv

   apt-get install libopencv-core-dev libopencv-highgui-dev \
               libopencv-calib3d-dev libopencv-features2d-dev \
               libopencv-imgproc-dev libopencv-video-dev

OpenCV 2 headers not found on Arch Linux with OpenCV 4 #496 follow https://github.com/google/mediapipe/issues/496

git clone git@github.com:xiong-jie-y/mediapipe.git
cd mediapipe
git checkout add_gpu_support_for_python
python setup.py gen_protos && python setup.py install --link-opencv

[Programmerwyl]

(mediapipe_gpu) wyl@wyl-MS-7C75:~/project/mediapipe_lib/evs$ python3 face_mesh.py WARNING: Logging before InitGoogleLogging() is written to STDERR I20220512 15:23:21.087352 80483 calculator_graph.cc:123] Initialize the GPU. libEGL warning: DRI2: failed to authenticate I20220512 15:23:21.102533 80483 gl_context_egl.cc:84] Successfully initialized EGL. Major : 1 Minor: 5 I20220512 15:23:21.108314 80549 gl_context.cc:334] GL version: 3.2 (OpenGL ES 3.2 Mesa 21.2.6) I20220512 15:23:21.108449 80483 calculator_graph.cc:127] Succeeded get GPU INFO: Created TensorFlow Lite delegate for GPU. time 0.49283814430236816 time 0.13471460342407227 time 0.13336586952209473 time 0.05213427543640137 time 0.1558845043182373 time 0.09224867820739746 time 0.14461874961853027 time 0.11232209205627441 time 0.11921167373657227 time 0.11662983894348145

[google-ml-butler[bot]]

Closing as stale. Please reopen if you'd like to work on this further.

[bossebandowski]

why is this closed?

[sureshdagooglecom]

Hi @bossebandowski , MediaPipe is not trained for GPU support , hence we are closing this issue.

[ChimeSkov]

Hi @sureshdagooglecom How is that possible, when you reference using GPU in your own docs? https://google.github.io/mediapipe/getting_started/gpu_support.html

[xiong-jie-y]

@Programmerwyl

Hi,

I doublechecked the hand example in my environment and it successfully used GPU. In your output, strange error occurs, so maybe OpenGL version difference or something?

libEGL warning: DRI2: failed to authenticate

I'm not sure if it's the reson. My libegl version is

libegl-mesa0/focal-updates,focal-security,now 21.2.6-0ubuntu0.1~20.04.2 amd64 [installed,automatic]
  free implementation of the EGL API -- Mesa vendor library

output of a hand example in my environment

ARNING: Logging before InitGoogleLogging() is written to STDERR
I20220523 19:14:59.307579 556588 calculator_graph.cc:123] Initialize the GPU.
I20220523 19:14:59.313134 556588 gl_context_egl.cc:84] Successfully initialized EGL. Major : 1 Minor: 5
I20220523 19:14:59.332114 556636 gl_context.cc:334] GL version: 3.2 (OpenGL ES 3.2 NVIDIA 470.129.06)
I20220523 19:14:59.332154 556588 calculator_graph.cc:127] Succeeded get GPU
INFO: Created TensorFlow Lite delegate for GPU.

[lghasemzadeh]

Hi @bossebandowski , MediaPipe is not trained for GPU support , hence we are closing this issue.

@sureshdagooglecom ok, then how about c++? can we run codes in c++ using GPU? BTW I asked in several issues that, is it possible to run on GPU and CPU, and your colleague said it is possible to run mediapipe on both GPU and CPU (anyone you want). then which of your answer is the final and correct answer?

[lghasemzadeh]

I successfully run some of the examples(facemesh, face detection, pose, holistic, hand) in python on GPU in recent mediapipe slightly_smiling_face The code is in this branch and the procedure to install the python module so far is

git clone git@github.com:xiong-jie-y/mediapipe.git
cd mediapipe
git checkout add_gpu_support_for_python
python setup.py gen_protos && python setup.py install --link-opencv

Thanks for the comment @xiong-jie-y. I checked lots of issues that says it is not possible to run mediapipe python on GPU. Even you would be able to do modification, mediapipe still uses CPU. Is that correct? Can you verify if it really uses GPU? Did you get any improvement in terms of speed/FPS/performance after changing it from CPU to GPU?

[xiong-jie-y]

@lghasemzadeh

Even you would be able to do modification, mediapipe still uses CPU. Is that correct?

It works in my environemt and I verified it by measuring FPS and seeing nvidia-smi :slightly_smiling_face:

[lghasemzadeh]

@lghasemzadeh

Even you would be able to do modification, mediapipe still uses CPU. Is that correct?

It works in my environemt and I verified it by measuring FPS and seeing nvidia-smi slightly_smiling_face

Thanks @xiong-jie-y I need to run my python codes on GPU both on my desktop and also on jetson nano. my questions are:

1. can I follow the instruction you provided, for jetson nano as well?
2. does the instruction work for previous version of the mediapie?

[bossebandowski]

Hi @bossebandowski , MediaPipe is not trained for GPU support , hence we are closing this issue.

Hey @sureshdagooglecom,

what does this mean? Are you suggesting that this is a datatype issue?

I do not understand why the website states "MediaPipe can work with TensorFlow to perform GPU inference on video cards that support CUDA." if that was the case.

To me it seems like the python API is the issue here rather than the model or its training.

[lghasemzadeh]

@bossebandowski @xiong-jie-y please check this issue and share your toughts/experience #3353

[Choyon10]

Hello @Programmerwyl do you able to resolve the libEGL warning: DRI2: failed to authenticate warning? I'm facing the same problem. Maybe, for this reason, GPU is not used during the execution time.

[kylemcdonald]

I followed through the instructions from @Programmerwyl plus one more small change: I had to modify my .bazelversion from 5.0.0 to 5.2.0. However I get this error when I try to instantiate a pose landmark model:

WARNING: Logging before InitGoogleLogging() is written to STDERR
I20221103 22:54:19.336287 172546 calculator_graph.cc:123] Initialize the GPU.
I20221103 22:54:19.348945 172546 gl_context_egl.cc:84] Successfully initialized EGL. Major : 1 Minor: 5
W20221103 22:54:19.348979 172546 gl_context_egl.cc:183] Creating a context with OpenGL ES 3 failed: UNKNOWN: ; eglChooseConfig() returned no matching EGL configuration for RGBA8888 D16 ES3 request. 
W20221103 22:54:19.348984 172546 gl_context_egl.cc:184] Fall back on OpenGL ES 2.
E20221103 22:54:19.349014 172546 calculator_graph.cc:132] UNKNOWN: ; eglChooseConfig() returned no matching EGL configuration for RGBA8888 D16 ES2 request. 
I20221103 22:54:19.349644 172546 gl_context_egl.cc:84] Successfully initialized EGL. Major : 1 Minor: 5
W20221103 22:54:19.349659 172546 gl_context_egl.cc:183] Creating a context with OpenGL ES 3 failed: UNKNOWN: ; eglChooseConfig() returned no matching EGL configuration for RGBA8888 D16 ES3 request. 
W20221103 22:54:19.349663 172546 gl_context_egl.cc:184] Fall back on OpenGL ES 2.
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Input In [5], in <cell line: 12>()
      9 mp_drawing = mp.solutions.drawing_utils
     10 mp_drawing_styles = mp.solutions.drawing_styles
---> 12 pose_detector = mp_pose.Pose(
     13     static_image_mode=True,
     14     model_complexity=2,
     15     enable_segmentation=False)

File ~/miniconda3/envs/gesture/lib/python3.9/site-packages/mediapipe/python/solutions/pose.py:146, in Pose.__init__(self, static_image_mode, model_complexity, smooth_landmarks, enable_segmentation, smooth_segmentation, min_detection_confidence, min_tracking_confidence)
    122 """Initializes a MediaPipe Pose object.
    123 
    124 Args:
   (...)
    143     https://solutions.mediapipe.dev/pose#min_tracking_confidence.
    144 """
    145 _download_oss_pose_landmark_model(model_complexity)
--> 146 super().__init__(
    147     binary_graph_path=_BINARYPB_FILE_PATH,
    148     side_inputs={
    149         'model_complexity': model_complexity,
    150         'smooth_landmarks': smooth_landmarks and not static_image_mode,
    151         'enable_segmentation': enable_segmentation,
    152         'smooth_segmentation':
    153             smooth_segmentation and not static_image_mode,
    154         'use_prev_landmarks': not static_image_mode,
    155     },
    156     calculator_params={
    157         'posedetectiongpu__TensorsToDetectionsCalculator.min_score_thresh':
    158             min_detection_confidence,
    159         'poselandmarkbyroigpu__tensorstoposelandmarksandsegmentation__ThresholdingCalculator.threshold':
    160             min_tracking_confidence,
    161     },
    162     outputs=['pose_landmarks', 'pose_world_landmarks', 'segmentation_mask'])

File ~/miniconda3/envs/gesture/lib/python3.9/site-packages/mediapipe/python/solution_base.py:289, in SolutionBase.__init__(self, binary_graph_path, graph_config, calculator_params, side_inputs, outputs, stream_type_hints)
    283   self._graph.observe_output_stream(stream_name, callback, True)
    285 self._input_side_packets = {
    286     name: self._make_packet(self._side_input_type_info[name], data)
    287     for name, data in (side_inputs or {}).items()
    288 }
--> 289 self._graph.start_run(self._input_side_packets)
    290 self.use_gpu = True

RuntimeError: Service "kGpuService", required by node posedetectiongpu__ImageToTensorCalculator, was not provided and cannot be created: ; eglChooseConfig() returned no matching EGL configuration for RGBA8888 D16 ES2 request. 

[hamaadtahiir]

@xiong-jie-y succesfully installed your solution but then got this error at the time of inference:

Major : 1 Minor: 5 I20230216 22:11:15.850900 21205 gl_context.cc:335] GL version: 3.2 (OpenGL ES 3.2 NVIDIA 515.43.04) I20230216 22:11:15.850953 21135 calculator_graph.cc:127] Succeeded get GPU 205 INFO: Created TensorFlow Lite delegate for GPU. Traceback (most recent call last): File "test_inference.py", line 59, in for i, data in enumerate(dataset, start=epoch_iter): File "/home/omnoai/anaconda3/envs/pf-afn/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 628, in next data = self._next_data() File "/home/omnoai/anaconda3/envs/pf-afn/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 671, in _next_data data = self._dataset_fetcher.fetch(index) # may raise StopIteration File "/home/omnoai/anaconda3/envs/pf-afn/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 58, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "/home/omnoai/anaconda3/envs/pf-afn/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 58, in data = [self.dataset[idx] for idx in possibly_batched_index] File "/media/omnoai/New Volume/python-webcam-flask/data/aligned_dataset_test_back.py", line 68, in getitem results = self.pose.process(I) File "/home/omnoai/anaconda3/envs/pf-afn/lib/python3.8/site-packages/mediapipe/python/solutions/pose.py", line 185, in process results = super().process(input_data={'image': image}) File "/home/omnoai/anaconda3/envs/pf-afn/lib/python3.8/site-packages/mediapipe/python/solution_base.py", line 379, in process self._get_packet_content(self._output_stream_type_info[stream_name], File "/home/omnoai/anaconda3/envs/pf-afn/lib/python3.8/site-packages/mediapipe/python/solution_base.py", line 577, in _get_packet_content return getattr(packetgetter, 'get' + packet_data_type.value)( AttributeError: module 'mediapipe.python.packet_getter' has no attribute 'get_gpu_buffer'

[HarshWinterBytes]

@xiong-jie-y succesfully installed your solution but then got this error at the time of inference:

Major : 1 Minor: 5 I20230216 22:11:15.850900 21205 gl_context.cc:335] GL version: 3.2 (OpenGL ES 3.2 NVIDIA 515.43.04) I20230216 22:11:15.850953 21135 calculator_graph.cc:127] Succeeded get GPU 205 INFO: Created TensorFlow Lite delegate for GPU. Traceback (most recent call last): File "test_inference.py", line 59, in for i, data in enumerate(dataset, start=epoch_iter): File "/home/omnoai/anaconda3/envs/pf-afn/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 628, in next data = self._next_data() File "/home/omnoai/anaconda3/envs/pf-afn/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 671, in _next_data data = self._dataset_fetcher.fetch(index) # may raise StopIteration File "/home/omnoai/anaconda3/envs/pf-afn/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 58, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "/home/omnoai/anaconda3/envs/pf-afn/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 58, in data = [self.dataset[idx] for idx in possibly_batched_index] File "/media/omnoai/New Volume/python-webcam-flask/data/aligned_dataset_test_back.py", line 68, in getitem results = self.pose.process(I) File "/home/omnoai/anaconda3/envs/pf-afn/lib/python3.8/site-packages/mediapipe/python/solutions/pose.py", line 185, in process results = super().process(input_data={'image': image}) File "/home/omnoai/anaconda3/envs/pf-afn/lib/python3.8/site-packages/mediapipe/python/solution_base.py", line 379, in process self._get_packet_content(self._output_stream_type_info[stream_name], File "/home/omnoai/anaconda3/envs/pf-afn/lib/python3.8/site-packages/mediapipe/python/solution_base.py", line 577, in _get_packet_content return getattr(packetgetter, 'get' + packet_data_type.value)( AttributeError: module 'mediapipe.python.packet_getter' has no attribute 'get_gpu_buffer'

I also encountered the same error reported, did you solve the problem?

[n-iwamoto]

Hi @xiong-jie-y ,

I've cloned & checkout out your branch that has the MediaPipe Python GPU changes and run the other commands successfully. However, I have a couple of questions:

1. where are the Python examples you were running to test GPU usage?
2. I found the .../mediapipe/python/solutions unit test apps, but what I was hoping to find was a Python example that calls the GPU versions of face mesh and face_detection, but would allow me to pass a .mp4 video file and generate an output .mp4 video file. Would you happen to have an example that does that?
3. the unit test apps appear to require a specific portrait.jpg -- I used a random face .jpg file and the face_mesh unit test failed. Do you know where I can get the portrait.jpg file the unit test seems to require?

Thanks for any help you could provide. Neil

[duchieuphan2k1]

I'm using mediapipe==0.10.9. For Ubuntu, it's simple to run the model on GPU:

• Firstly, install openGL:

sudo apt-get install mesa-common-dev libegl1-mesa-dev libgles2-mesa-dev sudo apt-get install mesa-utils

• Then, just add delegate=python.BaseOptions.Delegate.GPU to the BaseOptions initialization. Full code:

import mediapipe as mp from mediapipe.tasks import python from mediapipe.tasks.python import vision

base_options = python.BaseOptions(model_asset_path='mediapipe_models/pose_landmarker.task', delegate=python.BaseOptions.Delegate.GPU)

options = vision.PoseLandmarkerOptions(base_options=base_options, output_segmentation_masks=True) detector = vision.PoseLandmarker.create_from_options(options) mp_image = mp.Image.create_from_file('test.jpg') detector.detect(mp_image)

[n-iwamoto]

Thanks @duchieuphan2k1, appreciate the feedback; I'll give it a try.