ERROR while trying to access the Normalizedlandmarks from face mesh

[vamsee-asu2019]

I have been trying to access the landmarks from the face mesh . I have followed the steps given in https://github.com/google/mediapipe/issues/200#issuecomment-546619563 . I get the following error :

Compiling mediapipe/framework/formats/landmark.pb.cc [for host] failed: undeclared inclusion(s) in rule '//mediapipe/framework/formats:landmark_cc_proto': this rule is missing dependency declarations for the following files included by 'mediapipe/framework/formats/landmark.pb.cc': 'mediapipe/framework/formats/landmark.pb.h' Target //mediapipe/examples/desktop/face_mesh:face_mesh_cpu failed to build

OS : Windows 10

So while looking into BUILD file related to landmark_proto, I see that there are no dependencies mentioned (reference: https://github.com/google/mediapipe/blob/a92cff7a60031f5c3097b06e74416732d85b5011/mediapipe/framework/formats/BUILD#L250 ). Is there something that this file is missing in terms of mentioning the dependencies for landamrk_proto??

Could anyone help me with this?

[jiuqiant]

I can successfully build it bybazel build -c opt --action_env PYTHON_BIN_PATH="C://Path//to//python.exe" --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/framework/formats:landmark_cc_proto. What's the issue? Can you list all your changes?

[vamsee-asu2019]

@jiuqiant I have made the following changes : Created a new file demo_run_graph_main_landmarks.cc inside mediapipe\mediapipe\examples\desktop. This file is a replication of demo_run_graph_main.cc only difference is that I have the code to get the landmarks . I have included the followig headers: `#include "mediapipe/calculators/util/landmarks_to_render_data_calculator.pb.h"

include "mediapipe/framework/formats/landmark.pb.h"`

And then added a variable to tap landmarks : `constexpr char kDetectionsStream[] = "multi_face_landmarks";'

And then included the code to poll the output packet with landmarks as follows:

ASSIGN_OR_RETURN(mediapipe::OutputStreamPoller poller_detection, graph.AddOutputStreamPoller(kDetectionsStream));

Got the landmarks from the packet using :

auto& output_landmarks = detection_packet.Get<std::vector<::mediapipe::NormalizedLandmarkList>>();

And then trying to print the landmarks : for (const ::mediapipe::NormalizedLandmarkList& normalizedlandmarkList : output_landmarks) { std::cout << normalizedlandmarkList.DebugString(); }

After which I added the following lines to BUILD file inside mediapipe\mediapipe\examples\desktop

cc_library( name = "demo_run_graph_main_out", srcs = ["demo_run_graph_main_landmarks.cc"], deps = [ "//mediapipe/calculators/util:landmarks_to_render_data_calculator", "//mediapipe/framework:calculator_framework", "//mediapipe/framework/formats:image_frame", "//mediapipe/framework/formats:image_frame_opencv", "//mediapipe/framework/formats:landmark_cc_proto", "//mediapipe/framework/port:commandlineflags", "//mediapipe/framework/port:file_helpers", "//mediapipe/framework/port:opencv_highgui", "//mediapipe/framework/port:opencv_imgproc", "//mediapipe/framework/port:opencv_video", "//mediapipe/framework/port:parse_text_proto", "//mediapipe/framework/port:status",] )

And also I have changed the line:30 in BUILD file inside mediapipe\mediapipe\examples\desktop\face_mesh to : "//mediapipe/examples/desktop:demo_run_graph_main_out",

For the build I am using the following command :

`

[jiuqiant]

I copied your code into demo_run_graph_main.cc and compiled mediapipe/examples/desktop/face_mesh:face_mesh_cpu. It went well on a surface pro:

The diff is the following. Note that I didn't check the correctness of the code and just copied your code into the files.

diff --git a/mediapipe/examples/desktop/BUILD b/mediapipe/examples/desktop/BUILD
index 7772e21..c1ec33e 100644
--- a/mediapipe/examples/desktop/BUILD
+++ b/mediapipe/examples/desktop/BUILD
@@ -48,6 +48,8 @@ cc_library(
+++ b/mediapipe/examples/desktop/BUILD
@@ -48,6 +48,8 @@ cc_library(
         "//mediapipe/framework/port:opencv_video",
         "//mediapipe/framework/port:parse_text_proto",
         "//mediapipe/framework/port:status",
+        "//mediapipe/calculators/util:landmarks_to_render_data_calculator",
+        "//mediapipe/framework/formats:landmark_cc_proto",
     ],
 )

diff --git a/mediapipe/examples/desktop/demo_run_graph_main.cc b/mediapipe/examples/desktop/demo_run_graph_main.cc
index e09d2ab..57ed708 100644
--- a/mediapipe/examples/desktop/demo_run_graph_main.cc
+++ b/mediapipe/examples/desktop/demo_run_graph_main.cc
@@ -25,6 +25,8 @@
 #include "mediapipe/framework/port/opencv_video_inc.h"
 #include "mediapipe/framework/port/parse_text_proto.h"
 #include "mediapipe/framework/port/status.h"
+#include "mediapipe/calculators/util/landmarks_to_render_data_calculator.pb.h"
+#include "mediapipe/framework/formats/landmark.pb.h"

 constexpr char kInputStream[] = "input_video";
 constexpr char kOutputStream[] = "output_video";
@@ -53,7 +55,7 @@ mediapipe::Status RunMPPGraph() {
   LOG(INFO) << "Initialize the calculator graph.";
   mediapipe::CalculatorGraph graph;
   MP_RETURN_IF_ERROR(graph.Initialize(config));
-
+
   LOG(INFO) << "Initialize the camera or load the video.";
   cv::VideoCapture capture;
   const bool load_video = !FLAGS_input_video_path.empty();
@@ -78,6 +80,8 @@ mediapipe::Status RunMPPGraph() {
   LOG(INFO) << "Start running the calculator graph.";
   ASSIGN_OR_RETURN(mediapipe::OutputStreamPoller poller,
                    graph.AddOutputStreamPoller(kOutputStream));
+  ASSIGN_OR_RETURN(mediapipe::OutputStreamPoller poller_detection,
+                   graph.AddOutputStreamPoller("multi_face_landmarks"));
   MP_RETURN_IF_ERROR(graph.StartRun({}));

   LOG(INFO) << "Start grabbing and processing frames.";
@@ -118,7 +122,14 @@ mediapipe::Status RunMPPGraph() {
     mediapipe::Packet packet;
     if (!poller.Next(&packet)) break;
     auto& output_frame = packet.Get<mediapipe::ImageFrame>();
-
+    mediapipe::Packet detection_packet;
+    if (!poller_detection.Next(&packet))
+      break;
+    auto &output_landmarks = detection_packet.Get<std::vector<::mediapipe::NormalizedLandmarkList>>();
+    for (const ::mediapipe::NormalizedLandmarkList &normalizedlandmarkList : output_landmarks)
+    {
+      std::cout << normalizedlandmarkList.DebugString();
+    }
     // Convert back to opencv for display or saving.
     cv::Mat output_frame_mat = mediapipe::formats::MatView(&output_frame);
     cv::cvtColor(output_frame_mat, output_frame_mat, cv::COLOR_RGB2BGR);

[vamsee-asu2019]

@jiuqiant Yeah this works. the build is successful now. Thanks for that, I was hooked up on this for the whole day. I am not sure what's wrong with the new file though. Now when try to run the face_mesh_cpu.exe by passing the corresponding calculator graph config file I get the following error : F20210312 16:41:07.562562 24428 packet.h:768] Packet::Get() failed: Expected a Packet of type: class std::vector<class mediapipe::NormalizedLandmarkList,class std::allocator<class mediapipe::NormalizedLandmarkList> >, but received an empty Packet.

Command used : "bazel-bin/mediapipe/examples/desktop/face_mesh/face_mesh_cpu.exe" --calculator_graph_config_file=mediapipe/graphs/face_mesh/face_mesh_desktop_live.pbtxt

In the pbtxt file I [assed, I can see that there is an output stream named "multi_face_landmarks" of type std::vector. I am not sure why it says it received empty packet . What can be the reasons for this ? Am I accessing it the wrong way ? Were you able to run the executable and print the landmarks?

[jiuqiant]

Yeah, because there is a bug in the code. Should be poller_detection.Next(&detection_packet) rather than poller_detection.Next(&packet).

    mediapipe::Packet detection_packet;
    if (!poller_detection.Next(&detection_packet))
      break;
    auto &output_landmarks = detection_packet.Get<std::vector<::mediapipe::NormalizedLandmarkList>>();
    for (const ::mediapipe::NormalizedLandmarkList &normalizedlandmarkList : output_landmarks)
    {
      std::cout << "FaceLandmarks:";
      std::cout << normalizedlandmarkList.DebugString();
    }

Then, you will get something like the following in the terminal:

BTW, you can use the MediaPipe Python PyPI package if it's not necessary to be c++.

[vamsee-asu2019]

@jiuqiant Yeah I just came here to mention that. I was able to figure it out. Anyways thanks man for the help :)

I want to use this in another c++ project . which is why I am trying to use C++ version of the mediapipe.

[jiuqiant]

Glad to help. Closing this issue now.

[Hrituraj202]

Yeah, because there is a bug in the code. Should be poller_detection.Next(&detection_packet) rather than poller_detection.Next(&packet).

    mediapipe::Packet detection_packet;
    if (!poller_detection.Next(&detection_packet))
      break;
    auto &output_landmarks = detection_packet.Get<std::vector<::mediapipe::NormalizedLandmarkList>>();
    for (const ::mediapipe::NormalizedLandmarkList &normalizedlandmarkList : output_landmarks)
    {
      std::cout << "FaceLandmarks:";
      std::cout << normalizedlandmarkList.DebugString();
    }

Then, you will get something like the following in the terminal:

BTW, you can use the MediaPipe Python PyPI package if it's not necessary to be c++.

I have been trying to get the iris_landmarks. Even after resolving everything the way it is recommended ... I am facing an issue. The console shows no error but screen just freezes with camera on but giving no response at all ... Please help me..

[Fima2003]

Yeah, because there is a bug in the code. Should be poller_detection.Next(&detection_packet) rather than poller_detection.Next(&packet).

    mediapipe::Packet detection_packet;
    if (!poller_detection.Next(&detection_packet))
      break;
    auto &output_landmarks = detection_packet.Get<std::vector<::mediapipe::NormalizedLandmarkList>>();
    for (const ::mediapipe::NormalizedLandmarkList &normalizedlandmarkList : output_landmarks)
    {
      std::cout << "FaceLandmarks:";
      std::cout << normalizedlandmarkList.DebugString();
    }

Then, you will get something like the following in the terminal: BTW, you can use the MediaPipe Python PyPI package if it's not necessary to be c++.

I have been trying to get the iris_landmarks. Even after resolving everything the way it is recommended ... I am facing an issue. The console shows no error but screen just freezes with camera on but giving no response at all ... Please help me..

Hello! I have the same issue, as you do. I am also trying to figure out for iris. Have you managed to solve it?