Bilateral Filter on Python API

[andrearosasco]

In the segmentation code, a comment says that it's possible to improve the result by applying a bilateral filter

# Draw selfie segmentation on the background image.
# To improve segmentation around boundaries, consider applying a joint
# bilateral filter to "results.segmentation_mask" with "image".

I've found this c++ code bilateral_filter_calculator.cc but I can't find it in the python APIs. Do you plan to add it? Can you suggest any external implementation?

[sureshdagooglecom]

Hi @andrearosasco , We have calculators in framework which can be used for multiple ways and please refer this document for more information.

[andrearosasco]

Hi, thanks for the reply. On that page you sent I see a bunch of c++ files but no python interface. Am I correct to assume there is no way to use this bilateral filter implementation from python?

I was able to locate this file in my environment

# Generated by the protocol buffer compiler.  DO NOT EDIT!
# source: mediapipe/calculators/image/bilateral_filter_calculator.proto

import sys
_b=sys.version_info[0]<3 and (lambda x:x) or (lambda x:x.encode('latin1'))
from google.protobuf import descriptor as _descriptor
from google.protobuf import message as _message
from google.protobuf import reflection as _reflection
from google.protobuf import symbol_database as _symbol_database
from google.protobuf import descriptor_pb2
# @@protoc_insertion_point(imports)

_sym_db = _symbol_database.Default()

from mediapipe.framework import calculator_pb2 as mediapipe_dot_framework_dot_calculator__pb2
mediapipe_dot_framework_dot_calculator__options__pb2 = mediapipe_dot_framework_dot_calculator__pb2.mediapipe_dot_framework_dot_calculator__options__pb2

DESCRIPTOR = _descriptor.FileDescriptor(
  name='mediapipe/calculators/image/bilateral_filter_calculator.proto',
  package='mediapipe',
  syntax='proto2',
  serialized_pb=_b('\n=mediapipe/calculators/image/bilateral_filter_calculator.proto\x12\tmediapipe\x1a$mediapipe/framework/calculator.proto\"\xa7\x01\n BilateralFilterCalculatorOptions\x12\x13\n\x0bsigma_color\x18\x01 \x01(\x02\x12\x13\n\x0bsigma_space\x18\x02 \x01(\x02\x32Y\n\x03\x65xt\x12\x1c.mediapipe.CalculatorOptions\x18\xc1\xef\xf4y \x01(\x0b\x32+.mediapipe.BilateralFilterCalculatorOptions')
  ,
  dependencies=[mediapipe_dot_framework_dot_calculator__pb2.DESCRIPTOR,])
_sym_db.RegisterFileDescriptor(DESCRIPTOR)

_BILATERALFILTERCALCULATOROPTIONS = _descriptor.Descriptor(
  name='BilateralFilterCalculatorOptions',
  full_name='mediapipe.BilateralFilterCalculatorOptions',
  filename=None,
  file=DESCRIPTOR,
  containing_type=None,
  fields=[
    _descriptor.FieldDescriptor(
      name='sigma_color', full_name='mediapipe.BilateralFilterCalculatorOptions.sigma_color', index=0,
      number=1, type=2, cpp_type=6, label=1,
      has_default_value=False, default_value=float(0),
      message_type=None, enum_type=None, containing_type=None,
      is_extension=False, extension_scope=None,
      options=None),
    _descriptor.FieldDescriptor(
      name='sigma_space', full_name='mediapipe.BilateralFilterCalculatorOptions.sigma_space', index=1,
      number=2, type=2, cpp_type=6, label=1,
      has_default_value=False, default_value=float(0),
      message_type=None, enum_type=None, containing_type=None,
      is_extension=False, extension_scope=None,
      options=None),
  ],
  extensions=[
    _descriptor.FieldDescriptor(
      name='ext', full_name='mediapipe.BilateralFilterCalculatorOptions.ext', index=0,
      number=255670209, type=11, cpp_type=10, label=1,
      has_default_value=False, default_value=None,
      message_type=None, enum_type=None, containing_type=None,
      is_extension=True, extension_scope=None,
      options=None),
  ],
  nested_types=[],
  enum_types=[
  ],
  options=None,
  is_extendable=False,
  syntax='proto2',
  extension_ranges=[],
  oneofs=[
  ],
  serialized_start=115,
  serialized_end=282,
)

DESCRIPTOR.message_types_by_name['BilateralFilterCalculatorOptions'] = _BILATERALFILTERCALCULATOROPTIONS

BilateralFilterCalculatorOptions = _reflection.GeneratedProtocolMessageType('BilateralFilterCalculatorOptions', (_message.Message,), dict(
  DESCRIPTOR = _BILATERALFILTERCALCULATOROPTIONS,
  __module__ = 'mediapipe.calculators.image.bilateral_filter_calculator_pb2'
  # @@protoc_insertion_point(class_scope:mediapipe.BilateralFilterCalculatorOptions)
  ))
_sym_db.RegisterMessage(BilateralFilterCalculatorOptions)

_BILATERALFILTERCALCULATOROPTIONS.extensions_by_name['ext'].message_type = _BILATERALFILTERCALCULATOROPTIONS
mediapipe_dot_framework_dot_calculator__options__pb2.CalculatorOptions.RegisterExtension(_BILATERALFILTERCALCULATOROPTIONS.extensions_by_name['ext'])

# @@protoc_insertion_point(module_scope)

but I don't think it's exposing any API. I also ended up here but again it describes how to create calculators in C++. I think all you can do with python is accessing the Solutions right?

[willanie]

Hi @andrearosasco , I sure hope you get some response to your request for a method to do bilateral filtering in python. I've been trying to reduce the excessive landmark jitter with Solutions via python but have not succeeded.

[willanie]

Hi @andrearosasco , In the link below you will find python code that uses the built-in opencv python method that provides bilateral filtering image processing. I did try the filtering in my solutions.hands python script but unfortunately the landmark jitter is still happening. [https://www.geeksforgeeks.org/python-bilateral-filtering/]

here are the lines of python code provided in the link

import cv2

Read the image.

img = cv2.imread('taj.jpg')

Apply bilateral filter with d = 15,

sigmaColor = sigmaSpace = 75.

bilateral = cv2.bilateralFilter(img, 15, 75, 75)

Save the output.

cv2.imwrite('taj_bilateral.jpg', bilateral)

[sureshdagooglecom]

Hi @andrearosasco , Python API is provided for tech demo and experimentation. We don't plan to add this advanced feature to the python solution API.
It is possible to incorporate the bilateral filtering into Python apps with some modifications to the solution calculator graphs.

[google-ml-butler[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

[google-ml-butler[bot]]

Closing as stale. Please reopen if you'd like to work on this further.

[salihmarangoz]

cv::jointBilateralFilter()