Head segmentation in real time?

[chris-hndz]

Hi, I am looking for a solution to achieve head segmentation (hair with face segmentation) in real time on a webcam video. What do you recommend to achieve this with mediapipe? What segmentation or mediapipe tracking methods would be useful to achieve this?

Thanks in advance!

[brucechou1983]

I don't think you could achieve this by just using MediaPipe public models. We've done something similar in my team. We basically train our own model and serve it by MediaPipe.

[LGuilhermeRodrigues]

I would try Selfie Segmentation to remove the background and then Face Mesh to cut the neck

[sureshdagooglecom]

Hi @chris-hndz , These models are not part of MediaPipe solutions currently and this is a feature request.

[chris-hndz]

Thanks for the responses and ideas. I understand this is more of an improvement proposal for Mediapipe.

[kuaashish]

@chris-hndz,

Image Segmenter should be able to do this with the right model. You should try https://storage.googleapis.com/mediapipe-models/image_segmenter/deeplab_v3/float32/1/deeplab_v3.tflite with MediaPipe tasks. Thank you

[chris-hndz]

@kuaashish Thanks for your idea.

We did some tests with the Image Segmentation online demo with the Multi-class selfie segmentation model and we obtained reasonably good results for the segmentation mask of the face and hair. Maybe if we put both masks together we can get the head segmentation mask. But it still has a notorious delay.

And when we tested the Face Landmark Detection online demo we got this:

Here we see that the face landmarks limit well the lower part of the head.

Is there any way to merge the face and hair segmentation masks, and considering also the cutting the lower part of the face mask according to the face landmarks of the lower part of the face?

Thank you for your answer!

[kuaashish]

@chris-hndz,

You can use Multi-class selfie segmentation model and combine mask 1 (hair) and 3 (face-skin) as guided below:

https://developers.google.com/mediapipe/solutions/vision/image_segmenter#multiclass-model

However, If you will be able to achieve the full head model that can be instructions to other users. Thank you

[chris-hndz]

@kuaashish

Thank you very much for your reply, and advice for combining hair and face-skin mask.

Is there any way to cut the lower part of the face mask according to the face landmarks of the lower part of the face?

Thank you in advance!

[kuaashish]

@chris-hndz,

There is no off-the-shelf solution to this requirement currently. Thank you

[github-actions[bot]]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[chris-hndz]

Thanks @kuaashish for your reply.

I understand that there is no off-the-shelf solution to this requirement. Could you perhaps give some guidance or an idea of what would be the way to achieve this please?

Thank you in advance!