Accessing hand landmarks and other information in new releases

Please make sure that this is a solution issue.

System information (Please provide as much relevant information as possible)

Have I written custom code (as opposed to using a stock example script provided in Mediapipe): Yes

OS Platform and Distribution (e.g., Linux Ubuntu 16.04, Android 11, iOS 14.4): Mac OS X 11.4

MediaPipe version: v0.8.6

Bazel version: 3.7.2

Solution (e.g. FaceMesh, Pose, Holistic): Hands

Programming Language and version ( e.g. C++, Python, Java): C++

Describe the expected behavior:

I am trying to access real-time information about hand landmarks, scaled hand landmarks, handedness, and hand rects. In one of the earlier issues (https://github.com/google/mediapipe/issues/200), there was a complete guide on how to get these. It worked well until the new version, currently, v0.8.6 introduced some new mechanisms (actually it was v0.8.0 or v0.8.1) of how the graphs and modules interact.

The point is that the approach I used before - modification of the output streams of the hand tracking graphs, creation of the output stream pollers, and retrieval with the packets .Get() method in the main() loop do not work anymore: I get endless waiting of the main loop and it sends no new output frames (becomes idle) when there are no hands in the input frame.

I suppose that the issue is connected to the use of the poller.Next() method and could be somehow resolved with the use of GateCalculators and PreviousLoopbackCalculators. But honestly, it turned out to be quite tricky for me to handle this new runtime logic and I am asking for help and for a piece of advice on how to solve this the most correct and convenient way.

Standalone code you may have used to try to get what you need :

Mediapipe Hands: Desktop example

All the tips from https://github.com/google/mediapipe/issues/200

If there is a problem, provide a reproducible test case that is the bare minimum necessary to generate the problem. If possible, please share a link to Colab/repo link /any notebook:

My fork with all the changes needed to reproduce the issue: https://github.com/izakharkin/mediapipe. The main file is mediapipe/examples/desktop/hand_tracking/hand_tracking_cpu_main.cc.

Build:

bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/hand_tracking:hand_tracking_cpu

Run:

GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_cpu

Other info / Complete Logs : Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached:

Thank you very much!

Here are two solutions you can try:

Solution 1. Try tool::AddMultiStreamCallback with settingobserve_timestamp_bounds to true. Here is the usage example: https://github.com/google/mediapipe/blob/a9b643e0f5978425d948050a2228398929733d05/mediapipe/framework/tool/sink_test.cc#L188 By doing this, when no hands are detected, the callback calculator sends out empty packets.

Solution 2. Patch the following to allow OutputStreamPoller to observe timestamp bounds.

diff --git a/mediapipe/framework/calculator_graph.cc b/mediapipe/framework/calculator_graph.cc
index d95d6d3..6cff78d 100644
--- a/mediapipe/framework/calculator_graph.cc
+++ b/mediapipe/framework/calculator_graph.cc
@@ -465,7 +465,7 @@
 }

 absl::StatusOr<OutputStreamPoller> CalculatorGraph::AddOutputStreamPoller(
-    const std::string& stream_name) {
+    const std::string& stream_name, bool observe_timestamp_bounds) {
   RET_CHECK(initialized_).SetNoLogging()
       << "CalculatorGraph is not initialized.";
   int output_stream_index = validated_graph_->OutputStreamIndex(stream_name);
@@ -479,7 +479,7 @@
       stream_name, &any_packet_type_,
       std::bind(&CalculatorGraph::UpdateThrottledNodes, this,
                 std::placeholders::_1, std::placeholders::_2),
-      &output_stream_managers_[output_stream_index]));
+      &output_stream_managers_[output_stream_index], observe_timestamp_bounds));
   OutputStreamPoller poller(internal_poller);
   graph_output_streams_.push_back(std::move(internal_poller));
   return std::move(poller);
diff --git a/mediapipe/framework/calculator_graph.h b/mediapipe/framework/calculator_graph.h
index fb0bb69..0e6d53b 100644
--- a/mediapipe/framework/calculator_graph.h
+++ b/mediapipe/framework/calculator_graph.h
@@ -164,7 +164,8 @@
   // polling API for accessing a stream's output. Should only be called before
   // Run() or StartRun(). For asynchronous output, use ObserveOutputStream. See
   // also the helpers in tool/sink.h.
-  StatusOrPoller AddOutputStreamPoller(const std::string& stream_name);
+  StatusOrPoller AddOutputStreamPoller(const std::string& stream_name,
+                                       bool observe_timestamp_bounds = false);

   // Gets output side packet by name after the graph is done. However, base
   // packets (generated by PacketGenerators) can be retrieved before
diff --git a/mediapipe/framework/graph_output_stream.cc b/mediapipe/framework/graph_output_stream.cc
index 6639bb8..8235d3c 100644
--- a/mediapipe/framework/graph_output_stream.cc
+++ b/mediapipe/framework/graph_output_stream.cc
@@ -125,9 +125,10 @@
 absl::Status OutputStreamPollerImpl::Initialize(
     const std::string& stream_name, const PacketType* packet_type,
     std::function<void(InputStreamManager*, bool*)> queue_size_callback,
-    OutputStreamManager* output_stream_manager) {
+    OutputStreamManager* output_stream_manager, bool observe_timestamp_bounds) {
   MP_RETURN_IF_ERROR(GraphOutputStream::Initialize(stream_name, packet_type,
-                                                   output_stream_manager));
+                                                   output_stream_manager,
+                                                   observe_timestamp_bounds));
   input_stream_handler_->SetQueueSizeCallbacks(queue_size_callback,
                                                queue_size_callback);
   return absl::OkStatus();
@@ -176,12 +177,16 @@
 bool OutputStreamPollerImpl::Next(Packet* packet) {
   CHECK(packet);
   bool empty_queue = true;
+  bool observed_timestamp_bound_change = false;
   Timestamp min_timestamp = Timestamp::Unset();
   mutex_.Lock();
   while (true) {
     min_timestamp = input_stream_->MinTimestampOrBound(&empty_queue);
+    observed_timestamp_bound_change =
+        input_stream_handler_->ProcessTimestampBounds() &&
+        prev_output_ts_ < min_timestamp.PreviousAllowedInStream();
     if (graph_has_error_ || !empty_queue ||
-        min_timestamp == Timestamp::Done()) {
+        min_timestamp == Timestamp::Done() || observed_timestamp_bound_change) {
       break;
     } else {
       handler_condvar_.Wait(&mutex_);
@@ -191,17 +196,26 @@
     mutex_.Unlock();
     return false;
   }
+  if (empty_queue) {
+    prev_output_ts_ = min_timestamp.PreviousAllowedInStream();
+  } else {
+    prev_output_ts_ = min_timestamp;
+  }
   mutex_.Unlock();
   if (min_timestamp == Timestamp::Done()) {
     return false;
   }
-  int num_packets_dropped = 0;
-  bool stream_is_done = false;
-  *packet = input_stream_->PopPacketAtTimestamp(
-      min_timestamp, &num_packets_dropped, &stream_is_done);
-  CHECK_EQ(num_packets_dropped, 0)
-      << absl::Substitute("Dropped $0 packet(s) on input stream \"$1\".",
-                          num_packets_dropped, input_stream_->Name());
+  if (!empty_queue) {
+    int num_packets_dropped = 0;
+    bool stream_is_done = false;
+    *packet = input_stream_->PopPacketAtTimestamp(
+        min_timestamp, &num_packets_dropped, &stream_is_done);
+    CHECK_EQ(num_packets_dropped, 0)
+        << absl::Substitute("Dropped $0 packet(s) on input stream \"$1\".",
+                            num_packets_dropped, input_stream_->Name());
+  } else if (observed_timestamp_bound_change) {
+    *packet = Packet().At(Timestamp(min_timestamp.PreviousAllowedInStream()));
+  }
   return true;
 }

diff --git a/mediapipe/framework/graph_output_stream.h b/mediapipe/framework/graph_output_stream.h
index 393407a..1110a10 100644
--- a/mediapipe/framework/graph_output_stream.h
+++ b/mediapipe/framework/graph_output_stream.h
@@ -143,7 +143,8 @@
   absl::Status Initialize(
       const std::string& stream_name, const PacketType* packet_type,
       std::function<void(InputStreamManager*, bool*)> queue_size_callback,
-      OutputStreamManager* output_stream_manager);
+      OutputStreamManager* output_stream_manager,
+      bool observe_timestamp_bounds = false);

   void PrepareForRun(std::function<void()> notification_callback,
                      std::function<void(absl::Status)> error_callback) override;
@@ -170,6 +171,7 @@
   absl::Mutex mutex_;
   absl::CondVar handler_condvar_ ABSL_GUARDED_BY(mutex_);
   bool graph_has_error_ ABSL_GUARDED_BY(mutex_);
+  Timestamp prev_output_ts_ ABSL_GUARDED_BY(mutex_) = Timestamp::Min();
 };

 }  // namespace internal

Remember to modify your hand_tracking_cpu_main.cc to have

auto status_or_poller =
      graph.AddOutputStreamPoller("foo", /*observe_timestamp_bounds=*/true);

google-ai-edge / mediapipe

Accessing hand landmarks and other information in new releases #2366