Virtual Backgrounds Not Rendering Properly: Seeking Guidance on WebRTC Implementation

[vidya-hub]

Issue Description:

I am currently working on a video calling application that incorporates virtual background support. I have implemented code from the tfjs branch and attempted to stream updated canvas data after segmentation in WebRTC streams.

Problem:

The segmented foreground mask is rendering correctly, but the background is appearing as a black color(Parent element BG color). I have experimented with various canvas operations, including applying the background using CSS styling and an HTMLImageElement.

Methods Tried:

Method 1 - CSS Styling:

  const bg = `url('${this.bg}') center center / cover`;
  this.canvas.style.background = bg;

Method 2 - HTMLImageElement:

const backgroundImage = new Image();
backgroundImage.src = "https://d..........................jpg";
backgroundImage.onload = () => {
    canvasRef.current.getContext("2d").drawImage(backgroundImage, 0, 0, 640, 480);
}

Observations: Despite these methods, the background is rendering as black (which is parent element background color) during streaming.

Request for Assistance: I am seeking guidance on resolving this issue. If anyone has encountered a similar problem or has insights into potential solutions, your input would be greatly appreciated.

[vidya-hub]

CodePen Reference

[vidya-hub]

I've fixed this issue using default canvas methods by creating canvas images from the foreground and background segmented output