search

PeterL1n / BackgroundMattingV2

Real-Time High-Resolution Background Matting
MIT License
6.85k stars 952 forks source link

Real-time background replacement in a web browser #72

Closed benbro closed 3 years ago

benbro commented 3 years ago

Is it possible to use it in a browser for real-time video background replacement? Are there instructions? Something like: https://ai.googleblog.com/2020/10/background-features-in-google-meet.html

swelcker commented 3 years ago

you basically just use a virtual (fake) webcam in the browser app, the background replacement reads a real webcam and feeds the virtual one, all outside of the browser task/thread/process

benbro commented 3 years ago

It will be a better UX to be able to do it in a web app without installing a separate virtual webcam app. Is it possible to use ONNX.js, tfjs or mediapipe to run the matting in the browser?

PeterL1n commented 3 years ago

I have attempted to try tfjs, but it is very slow and not usable. ONNX.js straight up can't be converted.

benbro commented 3 years ago

@PeterL1n can you please share your tfjs code and model? The Google Meet model is fast in the browser so we could probably make BackgroundMattingV2 model fast too.

PeterL1n commented 3 years ago

Export yourself using the TensorFlow Implementation: https://github.com/PeterL1n/BackgroundMattingV2-TensorFlow

benbro commented 3 years ago

Thanks