Remote video track not working with VirtualBackgroundProcessor

[Kstar0722]

It works VirtualBackgroundProcessor in local track, but I see blank screen in the remote track with VirtualBackgroundProcessor I checked remote video track object. and there's no processor(processor: undefined)

• [x] I have verified that the issue occurs with the latest twilio-video-processors.js release and is not marked as a known issue in the CHANGELOG.md.
• [x] I reviewed the Common Issues and open GitHub issues and verified that this report represents a potentially new issue.
• [x] I am not sharing any Personally Identifiable Information (PII) or sensitive account information (API keys, credentials, etc.) when reporting this issue.

Expected behavior:

Does not showing remote video track with background

Actual behavior:

Need to show remote video track with background

Software versions:

• [x] Browser(s): Chrome 103.0.5060.114 (Official Build) (arm64)
• [x] Operating System: macOS Monterey
• [x] twilio-video-processors.js: 1.0.2
• [x] Third-party libraries (e.g., Angular, React, etc.): Angular

[charliesantos]

Hey @Kstar0722 can you please provide steps to reproduce, code snippet on how you're adding the processor, and also a room sid?

[Kstar0722]

@charliesantos sorry it was my mistake, I guess. I published local track first and then add background lik

const track = await createLocalVideoTrack({
      deviceId: this.selectedVideoInputDevice
})

await this.currentRoom.localParticipant.publishTrack(track)

if (args.backgroundEnabled) {
      await this.addBackgroundToVideoTrack(track, args.backgroundUrl)
}

but the issue fixed I changed order like add background first and then publish the local track. like

const track = await createLocalVideoTrack({
      deviceId: this.selectedVideoInputDevice
})

if (args.backgroundEnabled) {
      await this.addBackgroundToVideoTrack(track, args.backgroundUrl)
}

await this.currentRoom.localParticipant.publishTrack(track)

but I wonder it should work if we publish track first and then add background. b/c when we need to toggle background (enable/disable), we have to just remove/add background processer in created track without create new track.

[charliesantos]

Hi @Kstar0722 , adding background after publishing track should work. Can you please provide a code snippet on how you're adding the background? That is, please show this.addBackgroundToVideoTrack(track, args.backgroundUrl) implementation. Also, are you getting any log messages? Can you please enable debug mode on twilio-video and provide logs?

[Kstar0722]

Hi @charliesantos , sure, addBackgroundToVideoTrack triggers attachBackgroundToTrack.

[charliesantos]

@Kstar0722 thank you for providing the code snippet. Nothing stands out to me so far. I have a couple of requests

1. Can you please enable debug mode on twilio-video and provide logs?
2. Please refactor your code in a way that you should initialize the video processors first so you can reuse them on demand later, especially if you have to toggle video on/off. Something like the following below. This is also the same pattern we implemented in our demo react app which works really great.

// Save this in your hooks or app state, or wherever your video track can access them
let blurProcessor;
let virtualBackgroundProcessor;

// Call this on your app start up
async function initializeVideoProcessors() {
  blurProcessor = new GaussianBlurBackgroundProcessor({...});
  virtualBackgroundProcessor = new VirtualBackgroundProcessor({...});
  await blurProcessor.loadModel();
  await virtualBackgroundProcessor.loadModel();
}

Now your addBackgroundToVideoTrack can be simpler

async function addBackgroundToVideoTrack(track, backgroundUrl) {
  try {
    const backgroundImage = await loadImage(backgroundUrl);
    virtualBackgroundProcessor.backgroundImage = backgroundImage;
    track.addProcessor(virtualBackgroundProcessor);
  } catch {
    track.addProcessor(blurProcessor);
  }
}

async function loadImage(url) {
  return new Promise((resolve, reject) => {
    const image = new Image();
    image.onload = () => resolve(image);
    image.onerror = reject;
    image.src = url;
  });
}

[Kstar0722]

Thanks @charliesantos , I will try to update my code like above.

[Kstar0722]

Hi @charliesantos I try to update my code like your solution. but it's not working.

[charliesantos]

Can you please enable debug mode on twilio-video and provide logs? Can you also push up your changes to a repo so I can take a look?

[Kstar0722]

ok, I will try to enable debug mode and give you callSid, maybe tomorrow morning but I can't share my code since it's private.

[Kstar0722]

@charliesantos I checked it and seems the problem is on our end. I will close the ticket since there's no issue in this lib. thanks for your help.

[charliesantos]

@Kstar0722 I'm glad this is now resolved. What was the issue you found? It might help others in case they run into a similar issue.