search

Volcomix / virtual-background

Demo on adding virtual background to a live video stream in the browser
https://volcomix.github.io/virtual-background
Apache License 2.0
474 stars 122 forks source link

Update the tflite models and respective wasm #62

Closed arcinston closed 10 months ago

arcinston commented 10 months ago

Hey @Volcomix

i noticed you were using the january edition of selfie segmentation model

a newer version has been published at https://developers.google.com/mediapipe/solutions/vision/image_segmenter

i wanted to open a PR to update this myself and save time but i faced an error running the tflite inference tool this also could be the same reason why the github workflow to build the wasm is also failing

Volcomix commented 10 months ago

Hey @arcinston,

The segmentation model of this demo is from January 2021, so it's indeed quite outdated now. Some details about the reason here even though those details are also outdated as it seems like the latest models are now released under Apache 2.0 license.

The tflite inference tool doesn't build with the latest commits of tensorflow (I disabled the github workflow a few weeks ago) and I was struggling to understand why at a glance when I investigated.

Please note that I initially started this project when there wasn't any public WASM build of TFLite. Since then, there are now (at least) 2 runtimes maintained by TFJS and Mediapipe teams, which rely on more recent version of the segmentation model and which probably handle more edge cases than I did in this demo:

I would suggest you to take a look at these implementations and perhaps replace the segmentation part of the pipeline with them. I believe that would be more future proof for your projects.

arcinston commented 10 months ago

thanks a lot for this will really help

also Thank You for this repository helped me get started with building an npm package for this

will let you know when it's ready :)

Volcomix commented 10 months ago

Glad to hear that this work was helpful to you. Looking forward to checking out your work when it's ready 😉

benbro commented 10 months ago

Can you please explain what we should change to use the new model?

Volcomix commented 10 months ago

I haven't tried so I'm not sure if anything changed in the new model architecture but I think you could download selfie_segmenter.tflite and replace selfiesegmentation_mlkit-256x256-2021_01_19-v1215.f16.tflite from the models directory. Then the mlkit model filename should be replaced by the new one (selfie_segmenter).

Please note that this model should replace the ML Kit model, not Meet.

benbro commented 10 months ago

Thank you for the detailed explanation. I've used the landscape model. I didn't see any noticeable change from the previous model.

There is a lot of flickering around the head which is especially apparent when using a solid background image. Any suggestion how to improve it? I've tried both the cpu and gpu segmenters.

The mediapipe segmenter has LIVE_STREAM running mode. Do you know if it can improve the result when the subject is moving his head in a live video?

Volcomix commented 10 months ago

The only thing that comes to my mind is that if you use the landscape model, you'll need to change the resolution of the ML Kit model. Perhaps some visual issues could come from this. Playing with the post-processing sliders could also help to reduce the flickering around the head. I don't know much about the latest version of mediapipe segmenter so I can't share more insights about it :flushed:.

benbro commented 10 months ago

The flickering happens with all models and also in the demo. I'll open a new issue to discuss it.