iOS mediapipe pose tracking - remove blue background of the input video

[trunghvbk]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

Yes

OS Platform and Distribution

MacOS 13.6 iOS 17.1 Xcode 15.0

MediaPipe version

The latest version (v10.7)

Bazel version

bazel 6.3.2-homebrew

Solution

Pose

Programming Language and version

Swift 5

Describe the actual behavior

The background of the video is blue

Describe the expected behaviour

I want the background of the video is transparent to see the scene behind

I built the iOS app from the example code and it ran ok, but my question is there is a way to remove the blue background of the video when it's displayed pose detection?

• Folder to build iOS project: mediapipe/examples/ios/posetrackinggpu
• I only added some code to adjust the UI of the app (add buttons, labels), but not about changing the Mediapipe layer. The output image: https://imgur.com/a/lqQNyP4 Thanks!

Standalone code/steps you may have used to try to get what you need

No response

Other info / Complete Logs

[trunghvbk]

I see the color of input_image in the graph file is `{ r: 0 g: 0 b: 255 }

mediapipe/graphs/pose_tracking/subgraphs/pose_renderer_gpu.pbtxt

# Colors the segmentation mask with the color specified in the option.
node {
  calculator: "RecolorCalculator"
  input_stream: "IMAGE_GPU:input_image"
  input_stream: "MASK_GPU:segmentation_mask"
  output_stream: "IMAGE_GPU:segmented_image"
  node_options: {
    [type.googleapis.com/mediapipe.RecolorCalculatorOptions] {
      color { r: 0 g: 0 b: 255 }
      mask_channel: RED
      invert_mask: true
      adjust_with_luminance: false
    }
  }
}

And tried to change it to include alpha = 0 color { r: 0 g: 0 b: 255 a:0 } But it seems not to be supported, I build again and the error appears:

Message type "mediapipe.Color" has no field named "a"

[kuaashish]

Hi @trunghvbk,

We do not offer support for the legacy pose tracking solution as our primary focus is currently on the new Task APIs. We are actively providing Task APIs across various platforms, including iOS, Android, Python, C++, JavaScript, and customized Python solutions.

It's worth noting that using the "MaskOverlayCalculator" might be a potential solution. Issues such as Issue #1123 and Issue #4025 on GitHub can be helpful for background change or removal using the calculator. However, you will need to conduct your own investigation to implement these solutions for your specific use case. Unfortunately, at the moment, we have limited capabilities to address this further.

Thank you

[trunghvbk]

Hi @kuaashish Thanks for your response I understand, I tried and tested and found the solution for my need is to change the options like this:

node_options: {
    [type.googleapis.com/mediapipe.RecolorCalculatorOptions] {
      color { r: 0 g: 0 b: 255 }
      mask_channel: RED
      invert_mask: false
      adjust_with_luminance: true
    }
  }

The mask changes to overlay on the body instead of the background (invert_mask: false), and the mask color is a bit blur (adjust_with_luminance: true)

It's worth noting that using the "MaskOverlayCalculator" => OK I will invest in this way.

[kuaashish]

Hi @trunghvbk,

Could you please confirm whether you still require assistance in resolving this issue or if it has been resolved from your end? If now no longer issue from your end, We may close the issue and mark it as resolved.

Thank you!!

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No