search

jasonmayes / Real-Time-Person-Removal

Removing people from complex backgrounds in real time using TensorFlow.js in the web browser
Apache License 2.0
5.17k stars 651 forks source link

Could you please explain what models are used here #2

Closed golkir closed 4 years ago

golkir commented 4 years ago

This is a great experiment. Could you please elaborate a little bit on how you make this work? Thanks

MisakaMikoto128 commented 4 years ago

I also want to know

jasonmayes commented 4 years ago

As stated in the code comments it is the TensorFlow.js official bodypix model that does the body segmentation. (see https://github.com/tensorflow/tfjs-models/tree/master/body-pix)

You can see it in action here with my easy to get started tutorial on Glitch: https://glitch.com/~tensorflow-js-body-segmentation or Codepen: https://codepen.io/jasonmayes/pen/QWbNeJd

On top of this I then wrote my own custom algorithm to use that knowledge of body pixels to learn background over time as people move about the scene. I do some clean up too as the body pix model is not perfect - it will leave slight halo around the person where it classifies as not body when the pixel is part of body, so I use a sliding window effect to do rougher approximation to ensure no body is taken. If halo is left then whole image becomes that colour as person moves so that must be removed. This takes more processing time though, but I have an idea how to optimize that later.

b1xian commented 4 years ago

detection->crop->paste?

jasonmayes commented 4 years ago

detection -> sliding window to improve detection -> background update -> body removal

bebory commented 4 years ago

how do you perform "body removal"?