Hair segmentation with the new segmentation solutions

[nataliapam]

MediaPipe Solution (you are using)

Segmentation

Programming language

Python

Are you willing to contribute it

None

Describe the feature and the current behaviour/state

Will you include a tflite model for hair segmentation? It would be great because this way, together with facemesh, the whole head of a person could be detected, including the forehead which is not detected very accurately now. Thank you very much for your work

Will this change the current API? How?

It would be only an improvement

Who will benefit with this feature?

With this improvement the whole face will be detected allowing 100% control in all the head/face features (filters, virtual makeup, human expressions, hats, height measurement, people detection, hair designs....) Besides, this will increment the number of users of the previous hair detection feature (python users were not able to use it)

Please specify the use cases for this feature

Filters, virtual makeup, improve human expressions detection, hats, height measurement, people detection, hair designs....

Any Other info

THANKS A LOT, you are the BEST

[kuaashish]

Hello @nataliapam, Thank you for your interest in MediaPipe Solutions. Hair Segmentation is part of new image_segmenter solution as guided https://developers.google.com/mediapipe/solutions/vision/image_segmenter. However, Model should show hair as a different category https://storage.googleapis.com/mediapipe-tasks/image_segmenter/deeplabv3.tflite.

[nataliapam]

Hello @kuaashish, Thanks a lot for your response. When I run an inference following the example and with the deeplabv3.tflitemodel I found 21 categories. I suppose that maybe 20 categories are the ones from the standar deeplabv3 model (vehicles, household, animals, aeroplane, bicycle, boat, bus, car, motorbike, train, bottle, chair, dining table, potted plant, sofa, TV/monitor, bird, cat, cow, dog, horse, sheep, and person). Is the 21st category HAIR? When I see the 21 categories in the screen only 2 show something. One of the categories contains the person and another one the background. Am I doing something wrong? Thanks a lot again

`import mediapipe as mp from mediapipe.tasks import python from mediapipe.tasks.python import vision from mediapipe.python._framework_bindings import image from mediapipe.python._framework_bindings import image_frame import cv2 import math import numpy as np

DESIRED_HEIGHT = 480 DESIRED_WIDTH = 480

def resize_and_show(image): h, w = image.shape[:2] if h < w: img = cv2.resize( image, (DESIRED_WIDTH, math.floor(h/(w/DESIRED_WIDTH)))) else: img = cv2.resize( image, (math.floor(w/(h/DESIRED_HEIGHT)), DESIRED_HEIGHT)) cv2.imshow('img', img) cv2.waitKey(0) cv2.destroyAllWindows() cv2.waitKey(1)

BG_COLOR = (192, 192, 192) # gray MASK_COLOR = (255, 255, 255) # white

OutputType = vision.ImageSegmenterOptions.OutputType Activation = vision.ImageSegmenterOptions.Activation

Create the options that will be used for ImageSegmenter

model_path = 'models/deeplabv3.tflite' base_options = python.BaseOptions(model_asset_path=model_path)

options = vision.ImageSegmenterOptions(base_options=base_options, output_type=OutputType.CONFIDENCE_MASK)

Create the image segmenter

with vision.ImageSegmenter.create_from_options(options) as segmenter:

# Create the MediaPipe image file that will be segmented
image = mp.Image.create_from_file('images/womaniagenerated.png')

# Retrieve the masks for the segmented image
category_masks = segmenter.segment(image)

# Generate solid color images for showing the output segmentation mask.
image_data = image.numpy_view()
fg_image = np.zeros(image_data.shape, dtype=np.uint8)
fg_image[:] = MASK_COLOR
bg_image = np.zeros(image_data.shape, dtype=np.uint8)
bg_image[:] = BG_COLOR
for name in category_masks:

    condition = np.stack((name.numpy_view(),) * 3, axis=-1) > 0.2
    output_image = np.where(condition, fg_image, bg_image)
    resize_and_show(output_image)`

[Rameshv]

I tried downloading the hair segmentation model

!wget -O hair.tflite -q https://storage.googleapis.com/mediapipe-assets/hair_segmentation.tflite

and used during the inference like

base_options = BaseOptions(model_asset_path='hair.tflite')
options = vision.ImageSegmenterOptions(base_options=base_options,running_mode=VisionRunningMode.IMAGE,
                                              output_type=OutputType.CATEGORY_MASK)

I hoped this would have return the hair segmentations, but instead its throwing this error

The input tensor should have dimensions 1 x height x width x 3. Got 1 x 512 x 512 x 4.

So I suspect this is due to tensor size mismatch between the hair model and Vision image segmenter. But I could't find any references to that and stuck.

@nataliapam @kuaashish any ideas how to get around this??

[kuaashish]

Hello @Rameshv, Could you please raise a new issue with complete steps followed, complete error logs to look into the issue better. Thank you

[kuaashish]

Hi @schmidt-sebastian, Could you please look into this issue? Thank you

[Rameshv]

done @kuaashish https://github.com/google/mediapipe/issues/4266

[nataliapam]

Hello @kuaashish, Maybe I didn't understand what you told me

_"Thank you for your interest in MediaPipe Solutions. Hair Segmentation is part of new image_segmenter solution as guided https://developers.google.com/mediapipe/solutions/vision/image_segmenter. However, Model should show hair as a different category https://storage.googleapis.com/mediapipe-tasks/image_segmenter/deeplabv3.tflite."_

I understood that with the current model deeplabv3.tflite could be possible to find the hair in an image (now or in the next future) but maybe you wanted to say that I should find another model and use the image_segmenter with it? Please, could you clarify this? Should I create a model in tflite format?

Thanks a lot for your time and your patience and sorry for being so insistent.

[kuaashish]

@nataliapam,

We kindly request confirmation regarding the current status of the resolution of this issue. Specifically, we are inquiring whether this matter has been successfully resolved from your perspective, or if it remains unresolved. In the event of a successful resolution, we would appreciate your attention to this matter by reviewing the content of this thread and proceeding with the implementation of the hair segmentation example in Python, which has been integrated into our newly introduced Image Segmentation Task API. Thank you

[nataliapam]

I found the solution and you can close the issue. Mediapipe is great, thanks a lot

[kuaashish]

@nataliapam,

Thank you for your confirmation, We are now closing this issue and marking as resolved.