Closed Kasra-G closed 1 day ago
This does not happen often, approximately 2-5 times in a 200 image sequence that all produce detections, but it does mean that the empty packets cannot be relied on to determine if a detection failed.
As I mentioned in #552, I guess that it is caused because the input data is a sequence of static images while the graph is not running in static image mode. However, I don't have data that can reproduce this issue, so I'm not sure. Will you share the input data?
If you can't, then
Does it occur if you set use_prev_landmarks
to false
?
If you don't know how to do that, will you check if it occurs with the official sample code (C++)?
Hi! Thanks for looking into the issue. Per your first thought on reproducibility, I retested this on a fresh install of the plugin with only the code to detect when the empty packet has occurred. I ran the webcam source in Sync mode and in Non-blocking Sync mode (I have not tested Async mode). Both resulted in the poseLandmark stream eventually giving an empty packet before the correct one. It is interesting because the poseWorldLandmark stream does not do this (this behavior is the basis of the workaround) These are the steps I followed to reproduce the issue consistently:
if (poseLandmarks == null && poseWorldLandmarks != null)
{
r2 = TryGetNext(_poseLandmarksStream, out poseLandmarks, allowBlock, currentTimestampMicrosec);
}
Within 10 seconds or so (usually less), the breakpoint should trigger signaling that poseLandmark stream had erroneously sent an empty packet. I tried to determine if there were any poses that could cause the issue to occur more frequently but as far as I can tell it is totally random.
As for your suggestion of turning the use_prev_landmarks
setting to false
, maybe I will leave that to someone else... it looks like I would need another weekend or two of figuring out more mediapipe graphs to get that working.
I will take a look and see about testing it on the official C++ sample code, but since I did end up discovering a workaround, it has suddenly taken a lower priority compared to my other tasks. When I have gotten around to it and determined if it happens there, I'll notify. For now, I thought it best to address some of your points so that hopefully you or anyone else could reproduce the issue.
After digging into this issue, I suspect that the following is occurring (some Calculator names are normalized).
NOTE:
SwitchDemuxCalculator
will be triggered whenever its input is updated, but its container node won't be necessarily triggered (it is usually triggered only when all the inputs are given).input_video
at X.poselandmarkgpu__ImagePropertiesCalculator
adds its output to image_size
at X.image_size
packet triggers poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__SwitchDemuxCalculator
.PoseLandmarkByRoiGpu
adds its output to unfiltered_pose_landmarks
and unfiltered_auxiliary_landmarks
.
unfiltered_pose_landmarks
and unfiltered_auxiliary_landmarks
simultaneously, but the order does not matter here.unfiltered_pose_landmarks
packet triggers poselandmarkgpu__poselandmarkfiltering__switchcontainer_1__SwitchDemuxCalculator
poselandmarkgpu__poselandmarkfiltering__switchcontainer_1__SwitchDemuxCalculator
triggers poselandmarkgpu__poselandmarkfiltering__switchcontainer_1__VisibilitySmoothingCalculator_2
.poselandmarkgpu__poselandmarkfiltering__switchcontainer_1__VisibilitySmoothingCalculator_2
adds its output to filtered_visibility
.filtered_visibility
packet triggers poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__SwitchDemuxCalculator
.poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__SwitchDemuxCalculator
relays filtered_visibility
to poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__LandmarksSmoothingCalculator_2
.unfiltered_auxiliary_landmarks
packet triggers poselandmarkgpu__poselandmarkfiltering__LandmarksToDetectionCalculator
poselandmarkgpu__poselandmarkfiltering__LandmarksToDetectionCalculator
adds its output to aux_detection
.aux_detection
packet triggers poselandmarkgpu__poselandmarkfiltering__AlignmentPointsRectsCalculator
.poselandmarkgpu__poselandmarkfiltering__AlignmentPointsRectsCalculator
adds its output to roi
.roi
packet triggers poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__SwitchDemuxCalculator
.poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__SwitchDemuxCalculator
relays roi
to poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__LandmarksSmoothingCalculator_2
.poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__LandmarksSmoothingCalculator_2
is triggered.poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__LandmarksSmoothingCalculator_2
triggers poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__SwitchMuxCalculator
.poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__SwitchMuxCalculator
relays its input to filtered_landmarks
(== pose_landmarks
).The point is that Step B and Step C is concurrent.
As long as filtered_visibility
is calculated and relayed to LandmarksSmoothingCalculator
before roi
is calculated (or vice versa), this issue does not occur.
However, if filtered_visibility
is calculated and roi
is calculated before filtered_visibility
is relayed to LandmarksSmoothingCalculator
, the following occurs.
poselandmarkgpu__poselandmarkfiltering__switchcontainer_1__VisibilitySmoothingCalculator_2
adds its output to filtered_visibility
(Step B # 3).InputStreamHandler
pops the new filtered_visibility
packet from the queue and next_timestamp_bound_
for filtered_visibility
is updated.filtered_visibility
packet is sent to the corresponding input streams.poselandmarkgpu__poselandmarkfiltering__AlignmentPointsRectsCalculator
adds its output to roi
(Step C # 4).InputStreamHandler
pops the new roi
packet from the queue and next_timestamp_bound_
for roi
is updated.poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__SwitchDemuxCalculator
is triggered by the new filtered_visibility
packet.poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__SwitchDemuxCalculator
relays each input packet to poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__LandmarksSmoothingCalculator_2
.
image_size
is left as it is since it's empty.filtered_visibility
is relayed to the corresponding output stream.roi
is empty but the next_timestamp_bound_
was updated (at the 5th step), so the timestamp bound of the corresponding output stream is also updated.poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__LandmarksSmoothingCalculator_2
is triggered while the roi
is empty.poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__LandmarksSmoothingCalculator_2
does not output, but updates its output timestamp bound.poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__SwitchMuxCalculator
updates the timestamp bound of filtered_landmarks
.OutputStreamPoller#Next
), this change of the timestamp bound is detected and an empty packet is returned to the client.Note that if the 5th step occurs after the 7th step finishes, the timestamp bound of roi
won't be updated in the 7th step and poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__LandmarksSmoothingCalculator_2
won't be triggered in the 8th step.
This issue does not occur for world_landmarks
(filtered_world_landmarks
) because the corresponding SwitchContainer
has only one input stream and its container node will be triggered at most once.
After all, this issue occurs because SwitchDemuxCalculator
triggers LandmarksSmoothingCalculator
before all the dependent inputs are given.
I think we can avoid it by using DefaultInputStreamHandler
for the SwitchDemuxCalculator
instead of ImmediateInputStreamHandler
.
iff --git a/WORKSPACE b/WORKSPACE
index 28f4a6b..c568c90 100644
--- a/WORKSPACE
+++ b/WORKSPACE
@@ -63,6 +63,7 @@ http_archive(
"@//third_party:mediapipe_visibility.diff",
"@//third_party:mediapipe_model_path.diff",
"@//third_party:mediapipe_extension.diff",
+ "@//third_party:mediapipe_debug.diff",
# "@//third_party:mediapipe_emscripten_patch.diff",
],
sha256 = "6b43a4304ca4aa3a698906e4b4ff696d698d0b788baffd8284c03632712b1020",
diff --git a/third_party/mediapipe_debug.diff b/third_party/mediapipe_debug.diff
new file mode 100644
index 0000000..5b1074b
--- /dev/null
+++ b/third_party/mediapipe_debug.diff
@@ -0,0 +1,50 @@
+diff --git a/mediapipe/framework/tool/switch_container.proto b/mediapipe/framework/tool/switch_container.proto
+index a9c2d90..f38de33 100644
+--- a/mediapipe/framework/tool/switch_container.proto
++++ b/mediapipe/framework/tool/switch_container.proto
+@@ -27,4 +27,7 @@ message SwitchContainerOptions {
+
+ // Use DefaultInputStreamHandler for muxing & demuxing.
+ optional bool synchronize_io = 5;
++
++ optional bool synchronize_demux_io = 6;
++ optional bool synchronize_mux_io = 7;
+ }
+diff --git a/mediapipe/framework/tool/switch_demux_calculator.cc b/mediapipe/framework/tool/switch_demux_calculator.cc
+index 1c1d59d..7ea68d7 100644
+--- a/mediapipe/framework/tool/switch_demux_calculator.cc
++++ b/mediapipe/framework/tool/switch_demux_calculator.cc
+@@ -115,7 +115,7 @@ absl::Status SwitchDemuxCalculator::GetContract(CalculatorContract* cc) {
+ }
+ }
+ auto& options = cc->Options<mediapipe::SwitchContainerOptions>();
+- if (!options.synchronize_io()) {
++ if (!options.synchronize_io() && !options.synchronize_demux_io()) {
+ cc->SetInputStreamHandler("ImmediateInputStreamHandler");
+ }
+ cc->SetProcessTimestampBounds(true);
+diff --git a/mediapipe/framework/tool/switch_mux_calculator.cc b/mediapipe/framework/tool/switch_mux_calculator.cc
+index 9982ae4..a44bc00 100644
+--- a/mediapipe/framework/tool/switch_mux_calculator.cc
++++ b/mediapipe/framework/tool/switch_mux_calculator.cc
+@@ -154,7 +154,7 @@ absl::Status SwitchMuxCalculator::Process(CalculatorContext* cc) {
+ // Update the input channel index if specified.
+ channel_index_ = tool::GetChannelIndex(*cc, channel_index_);
+
+- if (options_.synchronize_io()) {
++ if (options_.synchronize_io() || options_.synchronize_mux_io()) {
+ // Start with adding input signals into channel_history_ and packet_history_
+ if (cc->Inputs().HasTag("ENABLE") &&
+ !cc->Inputs().Tag("ENABLE").IsEmpty()) {
+diff --git a/mediapipe/modules/pose_landmark/pose_landmark_filtering.pbtxt b/mediapipe/modules/pose_landmark/pose_landmark_filtering.pbtxt
+index bb3665f..27d141c 100644
+--- a/mediapipe/modules/pose_landmark/pose_landmark_filtering.pbtxt
++++ b/mediapipe/modules/pose_landmark/pose_landmark_filtering.pbtxt
+@@ -100,6 +100,7 @@ node {
+ options: {
+ [mediapipe.SwitchContainerOptions.ext] {
+ enable: true
++ synchronize_demux_io: true
+ contained_node: {
+ calculator: "LandmarksSmoothingCalculator"
+ options: {
@Kasra-G Will you test if the above patch resolves your issue?
Workaround
After all, this issue occurs because
SwitchDemuxCalculator
triggersLandmarksSmoothingCalculator
before all the dependent inputs are given. I think we can avoid it by usingDefaultInputStreamHandler
for theSwitchDemuxCalculator
instead ofImmediateInputStreamHandler
.iff --git a/WORKSPACE b/WORKSPACE index 28f4a6b..c568c90 100644 --- a/WORKSPACE +++ b/WORKSPACE @@ -63,6 +63,7 @@ http_archive( "@//third_party:mediapipe_visibility.diff", "@//third_party:mediapipe_model_path.diff", "@//third_party:mediapipe_extension.diff", + "@//third_party:mediapipe_debug.diff", # "@//third_party:mediapipe_emscripten_patch.diff", ], sha256 = "6b43a4304ca4aa3a698906e4b4ff696d698d0b788baffd8284c03632712b1020", diff --git a/third_party/mediapipe_debug.diff b/third_party/mediapipe_debug.diff new file mode 100644 index 0000000..5b1074b --- /dev/null +++ b/third_party/mediapipe_debug.diff @@ -0,0 +1,50 @@ +diff --git a/mediapipe/framework/tool/switch_container.proto b/mediapipe/framework/tool/switch_container.proto +index a9c2d90..f38de33 100644 +--- a/mediapipe/framework/tool/switch_container.proto ++++ b/mediapipe/framework/tool/switch_container.proto +@@ -27,4 +27,7 @@ message SwitchContainerOptions { + + // Use DefaultInputStreamHandler for muxing & demuxing. + optional bool synchronize_io = 5; ++ ++ optional bool synchronize_demux_io = 6; ++ optional bool synchronize_mux_io = 7; + } +diff --git a/mediapipe/framework/tool/switch_demux_calculator.cc b/mediapipe/framework/tool/switch_demux_calculator.cc +index 1c1d59d..7ea68d7 100644 +--- a/mediapipe/framework/tool/switch_demux_calculator.cc ++++ b/mediapipe/framework/tool/switch_demux_calculator.cc +@@ -115,7 +115,7 @@ absl::Status SwitchDemuxCalculator::GetContract(CalculatorContract* cc) { + } + } + auto& options = cc->Options<mediapipe::SwitchContainerOptions>(); +- if (!options.synchronize_io()) { ++ if (!options.synchronize_io() && !options.synchronize_demux_io()) { + cc->SetInputStreamHandler("ImmediateInputStreamHandler"); + } + cc->SetProcessTimestampBounds(true); +diff --git a/mediapipe/framework/tool/switch_mux_calculator.cc b/mediapipe/framework/tool/switch_mux_calculator.cc +index 9982ae4..a44bc00 100644 +--- a/mediapipe/framework/tool/switch_mux_calculator.cc ++++ b/mediapipe/framework/tool/switch_mux_calculator.cc +@@ -154,7 +154,7 @@ absl::Status SwitchMuxCalculator::Process(CalculatorContext* cc) { + // Update the input channel index if specified. + channel_index_ = tool::GetChannelIndex(*cc, channel_index_); + +- if (options_.synchronize_io()) { ++ if (options_.synchronize_io() || options_.synchronize_mux_io()) { + // Start with adding input signals into channel_history_ and packet_history_ + if (cc->Inputs().HasTag("ENABLE") && + !cc->Inputs().Tag("ENABLE").IsEmpty()) { +diff --git a/mediapipe/modules/pose_landmark/pose_landmark_filtering.pbtxt b/mediapipe/modules/pose_landmark/pose_landmark_filtering.pbtxt +index bb3665f..27d141c 100644 +--- a/mediapipe/modules/pose_landmark/pose_landmark_filtering.pbtxt ++++ b/mediapipe/modules/pose_landmark/pose_landmark_filtering.pbtxt +@@ -100,6 +100,7 @@ node { + options: { + [mediapipe.SwitchContainerOptions.ext] { + enable: true ++ synchronize_demux_io: true + contained_node: { + calculator: "LandmarksSmoothingCalculator" + options: {
Where can I check that code? @homuler @Kasra-G Thank you..
@dayowoo The code no longer works because MediaPipe is updated. Can you still reproduce this issue in the latest version?
close it because the solution is replaced with the Task API.
Plugin Version or Commit ID
v0.10.0
Unity Version
2021.3.3f1
Your Host OS
Windows 10
Target Platform
Windows Standalone
Target Device
No response
[Windows Only] Visual Studio C++ and Windows SDK Version
Visual Studio C++: 14.29.30139.0 Windows SDK: 10.0.17763
[Linux Only] GCC/G++ and GLIBC Version
No response
[Android Only] Android Build Tools and NDK Version
No response
[iOS Only] XCode Version
No response
Build Command
python build.py build --desktop cpu -vv
Bug Description
In the documentation, it says that setting observeTimestampBounds to true will generate an empty packet if mediapipe does not find a detection. However, I noticed that for some streams (poseLandmarkStream in PoseTrackingSolution, there could be more in other solutions), an empty packet is prepended to the detection packet, seemingly at random. This does not happen often, approximately 2-5 times in a 200 image sequence that all produce detections, but it does mean that the empty packets cannot be relied on to determine if a detection failed.
Steps to Reproduce the Bug
Switch off observeTimestampBounds on the poseLandmarksStream to false and this no longer happens.
Log
Unfortunately the log file contains sensitive information, so I am not permitted to share.
Screenshot/Video
No response
Additional Context
Workaround