search

livekit / track-processors-js

Pre-built track processors for background images, blur, etc for use with the LiveKit JS Client SDK
https://livekit.io
Apache License 2.0
34 stars 15 forks source link

Adopt WebGL for virtual background when available #44

Open kyleparrott opened 3 months ago

kyleparrott commented 3 months ago

I've been migrating my app from Twilio and recently had some users reporting performance issues, which appear to be related to using virtual backgrounds in our livekit calls.

I was able to reproduce this on the track processor sample app when applying a cpu slowdown in the chrome dev tools. When using the same cpu slowdown settings on a Twilio virtual background example app I did not see the same long running tasks in performance profiles. I've attached screenshots from the profiles for the Livekit and Twilio example apps below.

The performance difference looks like it comes down to Twilio's library using a webgl pipeline to apply the virtual background when webgl is supported.

Twilio profile: image

Livekit profile: image

lebaudantoine commented 4 weeks ago

Additionally, by utilizing WebGL2 like Twilio, we can achieve higher-quality effects that align more closely with the methods described by the Google Meet team in their 2020 papers. You can read more about their approach here: Google Research Blog.

The mask is then used to render the video output via WebGL2, with the background blurred or replaced.

For further insights, check out the Rendering Effects section, where they detail their workaround. They mention a Bilateral filter which seems to have been implemented in WebGL by Twilio here