search

jeeliz / jeelizFaceFilter

Javascript/WebGL lightweight face tracking library designed for augmented reality webcam filters. Features : multiple faces detection, rotation, mouth opening. Various integration examples are provided (Three.js, Babylon.js, FaceSwap, Canvas2D, CSS3D...).
https://jeeliz.com
Apache License 2.0
2.68k stars 535 forks source link

Ability to change background #116

Closed ghost closed 5 years ago

ghost commented 5 years ago

I have been searching for a javascript/react based library where we can change the background with a custom image without a green screen. I have found react-native based libraries but none for the web. Can you assist me with how do I approach this? Thanks

ghost commented 5 years ago

I am looking for something similar to this https://twitter.com/Baconbrix/status/1160736375118819328?s=20

xavierjs commented 5 years ago

Segmentation is a quite costly operation. We are not currently working on this and I don't have any solution.

I don't have seen any app working in the browser doing this properly.

Native app with a good background removal all use the depth field of the camera (often reconstructed by the hardware using 2 cameras and stereovision).

If you have a way to get the depth field into the browser than with jeeliz facefilter you can detect the face, compute the average value of the depth on the face and remove everything which is further. But I don't think this is possible and it would currently remain very device dependant. There is a draft of the W3C to get the depth field into the browser with MediaCapture API, but I think nothing will be ready before a few years (see https://www.w3.org/TR/mediacapture-depth/).

If you want to do something with specific hardware/kiosk you can use the Intel Realsense scanner (https://www.intelrealsense.com/) You can get the 16 bits depth field mapped over the red and green color channel with getUserMedia API.

ghost commented 5 years ago

Thank you so much for your reply. Do you think it's a better idea to just use a 3rd party application like Xsplit or chromacam and use that for replacing the background for the react app?

xavierjs commented 5 years ago

I don't know, these kinds of apps don't seem to work very well...

ghost commented 5 years ago

Got it. Thanks mate!