Closed arcinston closed 10 months ago
Hey @arcinston,
The segmentation model of this demo is from January 2021, so it's indeed quite outdated now. Some details about the reason here even though those details are also outdated as it seems like the latest models are now released under Apache 2.0 license.
The tflite inference tool doesn't build with the latest commits of tensorflow (I disabled the github workflow a few weeks ago) and I was struggling to understand why at a glance when I investigated.
Please note that I initially started this project when there wasn't any public WASM build of TFLite. Since then, there are now (at least) 2 runtimes maintained by TFJS and Mediapipe teams, which rely on more recent version of the segmentation model and which probably handle more edge cases than I did in this demo:
I would suggest you to take a look at these implementations and perhaps replace the segmentation part of the pipeline with them. I believe that would be more future proof for your projects.
thanks a lot for this will really help
also Thank You for this repository helped me get started with building an npm package for this
will let you know when it's ready :)
Glad to hear that this work was helpful to you. Looking forward to checking out your work when it's ready 😉
Can you please explain what we should change to use the new model?
I haven't tried so I'm not sure if anything changed in the new model architecture but I think you could download selfie_segmenter.tflite and replace selfiesegmentation_mlkit-256x256-2021_01_19-v1215.f16.tflite
from the models directory.
Then the mlkit
model filename should be replaced by the new one (selfie_segmenter
).
Please note that this model should replace the ML Kit
model, not Meet
.
Thank you for the detailed explanation. I've used the landscape model. I didn't see any noticeable change from the previous model.
There is a lot of flickering around the head which is especially apparent when using a solid background image. Any suggestion how to improve it? I've tried both the cpu and gpu segmenters.
The mediapipe segmenter has LIVE_STREAM running mode. Do you know if it can improve the result when the subject is moving his head in a live video?
The only thing that comes to my mind is that if you use the landscape model, you'll need to change the resolution of the ML Kit model. Perhaps some visual issues could come from this. Playing with the post-processing sliders could also help to reduce the flickering around the head. I don't know much about the latest version of mediapipe segmenter so I can't share more insights about it :flushed:.
Hey @Volcomix
i noticed you were using the january edition of selfie segmentation model
a newer version has been published at https://developers.google.com/mediapipe/solutions/vision/image_segmenter
i wanted to open a PR to update this myself and save time but i faced an error running the tflite inference tool this also could be the same reason why the github workflow to build the wasm is also failing