[Solution: Hand tracking] How to get landmarks on GPU mode?

[Kami-chanw]

When I use hand tracking solution of CPU version, I can get landmarks in this way: auto& output_landmarksVector = packet.Get<std::vector<mediapipe::NormalizedLandmarkList>>(); mediapipe::NormalizedLandmarkList output_landmarks = output_landmarksVector[0]; const mediapipe::NormalizedLandmark landmark = output_landmarks.landmark(8); float x = landmark.x() * camera_frame.cols; float y = landmark.y() * camera_frame.rows; But when I tried to use the same way to obtain landmarks, GLOG shows that Get() failed: The Packet stores "mediapipe::GpuBuffer", but "std::vector<mediapipe::NormalizedLandmarkList, std::allocator<mediapipe::NormalizedLandmarkList> >" was requested. So how to get landmarks from GpuBuffter?

[sureshdagooglecom]

Hi @Kami-chanw , Could you provide steps to reproduce this issue.

[Kami-chanw]

My procedures are as follows:

1. In order to use 'landmarks, I opened BUILD file in /mediapipe/examples/desktop, then added "//mediapipe/framework/formats:landmark_cc_proto", to demo_run_graph_main_gpu module.

2. I opened demo_run_graph_main_gpu.cc and added #include "mediapipe/framework/formats/landmark.pb.h", then modified the file like this:

   ...
   // Get the graph result packet, or stop if that fails.
   mediapipe::Packet packet;
   if (!poller.Next(&packet)) break;
   std::unique_ptr<mediapipe::ImageFrame> output_frame;
   
         auto& output_landmarksVector = packet.Get<std::vector<mediapipe::NormalizedLandmarkList>>();
         mediapipe::NormalizedLandmarkList output_landmarks = output_landmarksVector[0];
         const mediapipe::NormalizedLandmark landmark = output_landmarks.landmark(8);
         float x = landmark.x() * camera_frame.cols; 
         float y = landmark.y() * camera_frame.rows;
         float z = landmark.z();
         LOG(INFO) << x <<"  y:" << y << "  z:" << z;
   // Convert GpuBuffer to ImageFrame.
   MP_RETURN_IF_ERROR(gpu_helper.RunInGlContext(
       [&packet, &output_frame, &gpu_helper]() -> absl::Status {
         auto& gpu_frame = packet.Get<mediapipe::GpuBuffer>();
         auto texture = gpu_helper.CreateSourceTexture(gpu_frame);
   ...

3. I ran bazel build -c opt --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 mediapipe/examples/desktop/hand_tracking:hand_tracking_gpu in terminal.
4. It showed

   
   ...
   F20220401 00:52:01.706887 24889 packet.h:755] Packet::Get() failed: The Packet stores "mediapipe::GpuBuffer", but "std::vector<mediapipe::NormalizedLandmarkList, std::allocator<mediapipe::NormalizedLandmarkList> >" was requested.
   *** Check failure stack trace: ***
   @     0x55b58637d5ab  google::LogMessage::SendToLog()
   @     0x55b58637955d  google::LogMessage::Flush()
   @     0x55b5863799cd  google::LogMessageFatal::~LogMessageFatal()
   @     0x55b585c86628  RunMPPGraph()
   @     0x55b585c6e779  main
   @     0x7f963b8100b3  __libc_start_main
   @     0x55b585c8216e  _start
   @              (nil)  (unknown)

[wuheng0711]

I also encountered the same problem. Have you solved it?

[Kami-chanw]

Are you Chinese? If so, I will use Chinese. 你应该是中国人吧，我直接用中文了。

1. 你要把/mediapipe/examples/desktop/demo_run_graph_main_gpu.cc里面的char kOutputStream[] = "output_video"改成char kOutputStream[] = "hand_landmarks"
2. 把和图像处理的代码注释掉，只留下坐标处理代码。（图像和坐标似乎不能同时获取，如果另开一个poller会让程序速度急剧下降）
3. 注意增加对poller.QueueSize()的判断。

   ...
   // Get the graph result packet, or stop if that fails.
   mediapipe::Packet packet;
   if (!poller.Next(&packet)) break;
   if (poller.QueueSize() == 0) continue;
   auto& output_landmarksVector = packet.Get<std::vector<mediapipe::NormalizedLandmarkList>>();
   mediapipe::NormalizedLandmarkList output_landmarks = output_landmarksVector[0];
   const mediapipe::NormalizedLandmark landmark = output_landmarks.landmark(8);
   float x = landmark.x() * camera_frame.cols; 
   float y = landmark.y() * camera_frame.rows;
   float z = landmark.z();
   LOG(INFO) << x <<"  y:" << y << "  z:" << z;
   ...

[wuheng0711]

Are you Chinese? If so, I will use Chinese. 你应该是中国人吧，我直接用中文了。

1. 你要把/mediapipe/examples/desktop/demo_run_graph_main_gpu.cc里面的char kOutputStream[] = "output_video"改成char kOutputStream[] = "hand_landmarks"
2. 把和图像处理的代码注释掉，只留下坐标处理代码。（图像和坐标似乎不能同时获取，如果另开一个poller会让程序速度急剧下降）
3. 注意增加对poller.QueueSize()的判断。

...
// Get the graph result packet, or stop if that fails.
    mediapipe::Packet packet;
    if (!poller.Next(&packet)) break;
    if (poller.QueueSize() == 0) continue;
    auto& output_landmarksVector = packet.Get<std::vector<mediapipe::NormalizedLandmarkList>>();
    mediapipe::NormalizedLandmarkList output_landmarks = output_landmarksVector[0];
    const mediapipe::NormalizedLandmark landmark = output_landmarks.landmark(8);
    float x = landmark.x() * camera_frame.cols; 
    float y = landmark.y() * camera_frame.rows;
    float z = landmark.z();
    LOG(INFO) << x <<"  y:" << y << "  z:" << z;
...

Thank you very much. I have solved this problem. 非常感谢您，我已经解决了这个问题。 I created a poller of landmarks to get the coordinate points of the hand. 我创建了一个landmarks的poller，用来得到手部的坐标点。

1、在/mediapipe/examples/desktop中找到to demo_run_graph_main_gpu module，并添加"//mediapipe/framework/formats:landmark_cc_proto", "//mediapipe/framework/formats:detection_cc_proto", "//mediapipe/framework/formats:rect_cc_proto",

Linux only.

Must have a GPU with EGL support:

ex: sudo apt-get install mesa-common-dev libegl1-mesa-dev libgles2-mesa-dev

(or similar nvidia/amd equivalent)

cc_library( name = "demo_run_graph_main_gpu", srcs = ["demo_run_graph_main_gpu.cc"], deps = [ "//mediapipe/framework:calculator_framework", "//mediapipe/framework/formats:image_frame", "//mediapipe/framework/formats:image_frame_opencv", "//mediapipe/framework/formats:landmark_cc_proto", "//mediapipe/framework/formats:detection_cc_proto", "//mediapipe/framework/formats:rect_cc_proto", "//mediapipe/framework/port:file_helpers", "//mediapipe/framework/port:opencv_highgui", "//mediapipe/framework/port:opencv_imgproc", "//mediapipe/framework/port:opencv_video", "//mediapipe/framework/port:parse_text_proto", "//mediapipe/framework/port:status", "//mediapipe/gpu:gl_calculator_helper", "//mediapipe/gpu:gpu_buffer", "//mediapipe/gpu:gpu_shared_data_internal", "@com_google_absl//absl/flags:flag", "@com_google_absl//absl/flags:parse", ], ) 2、在demo_run_graph_main_gpu.cc添加

include "mediapipe/framework/formats/detection.pb.h"

include "mediapipe/framework/formats/landmark.pb.h"

include "mediapipe/framework/formats/rect.pb.h"

3、新建一个poller用来接收坐标点：constexpr char kOutputLandmarks[] = "hand_landmarks";

constexpr char kInputStream[] = "input_video"; constexpr char kOutputStream[] = "output_video"; constexpr char kOutputLandmarks[] = "hand_landmarks"; constexpr char kWindowName[] = "MediaPipe";

4、在ASSIGN_OR_RETURN(mediapipe::OutputStreamPoller poller, graph.AddOutputStreamPoller(kOutputStream));下一行，新增注册坐标点回调：

ASSIGN_OR_RETURN(mediapipe::OutputStreamPoller pPoller_landmarks, graph.AddOutputStreamPoller(kOutputLandmarks));

5、注释图像处理部分代码，并增加坐标点处理代码：

mediapipe::Packet packet; mediapipe::Packet packet_landmarks; std::vector hand_landmarks; hand_landmarks.clear(); if (!poller.Next(&packet)){ break; } printf("poller is not null.\n"); if(pPoller_landmarks.QueueSize() > 0){ if (pPoller_landmarks.Next(&packet_landmarks)){ std::vector output_landmarks = packet_landmarks.Get<std::vector>(); for (int m = 0; m < output_landmarks.size(); ++m){ mediapipe::NormalizedLandmarkList single_hand_NormalizedLandmarkList = output_landmarks[m]; for (int i = 0; i < single_hand_NormalizedLandmarkList.landmark_size(); ++i){ const mediapipe::NormalizedLandmark landmark = single_hand_NormalizedLandmarkList.landmark(i); int x = landmark.x() camera_frame.cols; int y = landmark.y() camera_frame.rows; hand_landmarks.push_back(x); hand_landmarks.push_back(y); } } printf("hand points size = %d.\n", hand_landmarks.size()); } }

[Kami-chanw]

ok，其实output stream poller可以直接删掉的

[wuheng0711]

确实，我按照您的方法将output stream poller删除后，速度得到了飞速提升。但是我在cpu版本下使用同样的方式，将注output stream poller注释，添加了landmarks后，出现了如下情况：1)处理速度没有得到提升2)得到的手部点与实时图像延迟了大概十秒钟请问你有遇到过这样的问题吗？发自我的手机-------- 原始邮件 --------发件人： Kami-chanw @.>日期： 2022年4月8日周五 晚上7:42收件人： google/mediapipe @.>抄送： wuheng0711 @.>, Comment @.>主 题： Re: [google/mediapipe] [Solution: Hand tracking] How to get landmarks on GPU mode? (Issue #3218)

ok，其实output stream poller可以直接删掉的

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

[bharathraja]

While running in CPU, with getting both image and landmark. I get weird delay when landmark is polled in addition to image. When i do post-processing based on landmark in the main "demo_run_graph_main.cc", i get high latency. But, when i do post-processing based on landmark inside "calc/utils/word_landmark_projection_calculator.cc" i get realtime latency. Why is there this difference of latency ?

I also tried adding "packet_presence_calculator" which gave wildly enormous delays, so avoiding using it.

[wuheng0711]

The steps of hand tracking are: palm detection + hand landmarks + tracking. When there is no hand, it is necessary to detect palm detection for each frame of image, which is time-consuming. When the hand is detected, tracking will be called in the next frame, which will reduce the time consumption.(hand tracking的步骤为：palm detection+hand landmarks+tracking。当没有目标时，需要每帧图像都检测palm detection，该步骤耗时较高。当检测到目标后，下一帧会调用tracking，会减少耗时。)发自我的手机-------- 原始邮件 --------发件人： Bharath Raja @.>日期： 2022年4月27日周三 晚上9:00收件人： google/mediapipe @.>抄送： wuheng0711 @.>, Comment @.>主 题： Re: [google/mediapipe] [Solution: Hand tracking] How to get landmarks on GPU mode? (Issue #3218) While running in CPU, with getting both image and landmark. I get weird delay in landmark output from no-pose to pose with high latency. The transition from pose to no-pose is at low latency.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

[Kami-chanw]

确实，我按照您的方法将output stream poller删除后，速度得到了飞速提升。但是我在cpu版本下使用同样的方式，将注output stream poller注释，添加了landmarks后，出现了如下情况：1)处理速度没有得到提升2)得到的手部点与实时图像延迟了大概十秒钟请问你有遇到过这样的问题吗？发自我的手机-------- 原始邮件 --------发件人： Kami-chanw @.>日期： 2022年4月8日周五 晚上7:42收件人： google/mediapipe @.>抄送： wuheng0711 @.>, Comment @.>主 题： Re: [google/mediapipe] [Solution: Hand tracking] How to get landmarks on GPU mode? (Issue #3218) ok，其实output stream poller可以直接删掉的 —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

CPU下我用的是其他方法。你现在问题解决了吗？如果没有的话我切去Ubuntu给你贴贴代码

[wuheng0711]

好的，非常感谢，麻烦您给我贴一下代码。在cpu下，我现在是同时获取点和图像解决了延迟的问题，但是还想提升一下速度。发自我的手机-------- 原始邮件 --------发件人： Kami-chanw @.>日期： 2022年4月27日周三 半夜11:23收件人： google/mediapipe @.>抄送： wuheng0711 @.>, Comment @.>主 题： Re: [google/mediapipe] [Solution: Hand tracking] How to get landmarks on GPU mode? (Issue #3218)

确实，我按照您的方法将output stream poller删除后，速度得到了飞速提升。但是我在cpu版本下使用同样的方式，将注output stream poller注释，添加了landmarks后，出现了如下情况：1)处理速度没有得到提升2)得到的手部点与实时图像延迟了大概十秒钟请问你有遇到过这样的问题吗？发自我的手机-------- 原始邮件 --------发件人： Kami-chanw @.>日期： 2022年4月8日周五 晚上7:42收件人： google/mediapipe @.>抄送： wuheng0711 @.>, Comment @.>主 题： Re: [google/mediapipe] [Solution: Hand tracking] How to get landmarks on GPU mode? (Issue #3218) ok，其实output stream poller可以直接删掉的 —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

CPU下我用的是其他方法。你现在问题解决了吗？如果没有的话我切去Ubuntu给你贴贴代码

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

[Kami-chanw]

好的，非常感谢，麻烦您给我贴一下代码。在cpu下，我现在是同时获取点和图像解决了延迟的问题，但是还想提升一下速度。发自我的手机-------- 原始邮件 --------发件人： Kami-chanw @.>日期： 2022年4月27日周三 半夜11:23收件人： google/mediapipe @.>抄送： wuheng0711 @.>, Comment @.>主 题： Re: [google/mediapipe] [Solution: Hand tracking] How to get landmarks on GPU mode? (Issue #3218) 确实，我按照您的方法将output stream poller删除后，速度得到了飞速提升。但是我在cpu版本下使用同样的方式，将注output stream poller注释，添加了landmarks后，出现了如下情况：1)处理速度没有得到提升2)得到的手部点与实时图像延迟了大概十秒钟请问你有遇到过这样的问题吗？发自我的手机-------- 原始邮件 --------发件人： Kami-chanw @.>日期： 2022年4月8日周五 晚上7:42收件人： google/mediapipe @.>抄送： wuheng0711 @.>, Comment @.>主 题： Re: [google/mediapipe] [Solution: Hand tracking] How to get landmarks on GPU mode? (Issue #3218) ok，其实output stream poller可以直接删掉的 —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.> CPU下我用的是其他方法。你现在问题解决了吗？如果没有的话我切去Ubuntu给你贴贴代码 —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.>

我看了一下，cpu这边也是和gpu那边一样的，除了output_video是landmarks而不是hand_landmarks。我觉得图像（就是mediapipe渲染的subGraph）和点应该是不能同时获得的

[bharathraja]

I solved it using timestamp bound option as "true" while setting polling as said in this issue #2366

[JasonJin8]

Are you Chinese? If so, I will use Chinese. 你应该是中国人吧，我直接用中文了。

1. 你要把/mediapipe/examples/desktop/demo_run_graph_main_gpu.cc里面的char kOutputStream[] = "output_video"改成char kOutputStream[] = "hand_landmarks"
2. 把和图像处理的代码注释掉，只留下坐标处理代码。（图像和坐标似乎不能同时获取，如果另开一个poller会让程序速度急剧下降）
3. 注意增加对poller.QueueSize()的判断。

...
// Get the graph result packet, or stop if that fails.
    mediapipe::Packet packet;
    if (!poller.Next(&packet)) break;
    if (poller.QueueSize() == 0) continue;
    auto& output_landmarksVector = packet.Get<std::vector<mediapipe::NormalizedLandmarkList>>();
    mediapipe::NormalizedLandmarkList output_landmarks = output_landmarksVector[0];
    const mediapipe::NormalizedLandmark landmark = output_landmarks.landmark(8);
    float x = landmark.x() * camera_frame.cols; 
    float y = landmark.y() * camera_frame.rows;
    float z = landmark.z();
    LOG(INFO) << x <<"  y:" << y << "  z:" << z;
...

I also resolved issue based on your suggestion Thanks