Virtual background processor leads to higher CPU usage

[Lisztos]

• [x] I have verified that the issue occurs with the latest twilio-video.js release and is not marked as a known issue in the CHANGELOG.md.
• [x] I reviewed the Common Issues and open GitHub issues and verified that this report represents a potentially new issue.
• [x] I verified that the Quickstart application works in my environment.
• [x] I am not sharing any Personally Identifiable Information (PII) or sensitive account information (API keys, credentials, etc.) when reporting this issue.

Code to reproduce the issue:

import { GaussianBlurBackgroundProcessor, VirtualBackgroundProcessor } from '@twilio/video-processors';
import { VideoTrack, AddProcessorOptions } from 'twilio-video';

const maskBlurRadius = 5;
const blurFilterRadius = 12;

const defaultProcessorOptions: AddProcessorOptions = {
  inputFrameBufferType: 'video',
  outputFrameBufferContextType: 'webgl2',
};

export default class VideoProcessor {
  processors = new VideoBackgroundProcessor();

  async getBlurBackground(blurLevel: number) {
    return this.processors.getGaussianBlurProcessor({
      maskBlurRadius,
      assetsPath: getBackgroundsPath(),
      blurFilterRadius: blurLevel,
    })
  }

  async blurVideoBackground(track: VideoTrack, blurLevel: number = blurFilterRadius) {
    const processor = await this.getBlurBackground(blurLevel);
    track.addProcessor(processor, defaultProcessorOptions);
  }
}

This is a snippet of our code, nonetheless, I follow standard implementation.

I created a performance profile with both the following setups to test the difference in CPU usage. In the test, I switched from no virtual background to blurred virtual background.

Current Setup:

• twilio-video: 2.26.2
• @twilio/video-processor: 1.0.2

With the current setup, you can see the CPU usage increase, having constant spikes. (screenshot 1)

Updated Setup:

• twilio-video: 2.27.0
• @twilio/video-processor: 2.0.0

After updating versions and switching to the Webgl2 pipeline, the CPU usage maxed out and stayed high. (screenshot 2)

Software versions:

• [x] Browser(s): Chrome 105.0.5195.127
• [x] Operating System: Windows 10 Pro
• [x] Twilio-video.js: 2.27.0
• [x] Twilio-video Processor: 2.0.0
• [x] Third-party libraries (e.g., Angular, React, etc.): Vue.js

[CyranMS]

@manjeshbhargav this is blocking us from upgrading

[anna-vasilko]

Hi @CyranMS, thank you for the detailed report. Our QE team is working to reproduce the performance degradation between mentioned versions. We will be setting priority of the issue based on their findings.

[anna-vasilko]

Hey all, the team here was able to reproduce this issue. We prioritize fixing it in this sprint.

[CyranMS]

That is great to hear and very fast. Thanks @anna-vasilko!

In the release note for the video processor v2 it said that webgl2 needs about 30% less cpu. Is that claim still accurate? We are upgrading mainly because of that benefit to have better meeting quality by using less cpu.

[manjeshbhargav]

@CyranMS , @Lisztos ,

I found out that the CPU max-out behavior (resulting in huge delays in the video stream) happens primarily on Intel-based Graphics Cards for 720p and greater resolutions. So, while we work to investigate further, I suggest a workaround:

1. addProcessor() - If video resolution is 720p,`LocalVideoTrack.restart()` at 480p if your app is running on Chrome with an Intel-based Graphics Card. Otherwise continue capture at 720p.
2. removeProcessor() - If original resolution was 720p, then `LocalVideoTrack.restart()` at 720p.

You can figure out the graphics card name by using the following code snippet:

/**
 * Get the graphics card information.
 * @returns {string} Graphics card information
 */
function getGraphicsCardInfo() {
  const gl = document.createElement('canvas').getContext('webgl2');
  if (!gl) {
    return {
      error: "no webgl",
    };
  }
  const debugInfo = gl.getExtension('WEBGL_debug_renderer_info');
  return gl.getParameter(debugInfo.UNMASKED_RENDERER_WEBGL);
}

[CyranMS]

Thanks for looking into it @manjeshbhargav! We'll check this out. Will there also be a proper fix any time soon?

[yura-oryshchych]

Hey @anna-vasilko @manjeshbhargav . Just wondering if anyone has an idea on when we can expect that fix. Really hoping this gets sorted out soon! We faced this issue a lot, cpu max-out behavior, delay in video streams Thanks

[CyranMS]

@anna-vasilko @manjeshbhargav any updates?

[juanjovazquez]

This is a huge blocker for us too. Same behavior.

[swapnonil-b]

Any updates on this? Performance is going to be important as we may have devices with low CPU capability.

[anna-vasilko]

Hi all, Apologies for the delay and the troubles caused by the CPU max-out issue. It was temporarily de-prioritized since the issue appeared to be less widespread than initially thought. Nevetherless please be assured that we are commited to address this, we aim to pick up this work in the next sprint. In the interim, I recommend using the workaround mentioned here to avoid the cpu maxout as the issue is specific to Intel-based graphic cards when video track resolution is 720p or greater.

[CyranMS]

Thanks for the update @anna-vasilko. Is there a rough estimate of when that will be done? This will help us to determine if we need to implement the workaround or if we wait for the new update.

[anna-vasilko]

Hey @CyranMS The work is in the current sprint. However it will likely spill over to the next one since the team have to prioritize the newly discovered iOS17 issues. Conservatively I would say this fix will ship 2-3 weeks from now.

[dmajor3]

Any update here @anna-vasilko ?

[BryanA2T]

Any update here @anna-vasilko? This is a crucial blocker for our use case.

[BryanA2T]

Any update here @manjeshbhargav?

[swapnonil-b]

Hopefully this is still being worked on? The set of devices is not as limited as we might have thought.

[ido-journey]

Here just to point out how critical this issue is, we just released the backgrounds feature for clients and already had a big percentage of them experiencing this, we had to disable it for now.

[alxppp]

An update would be great @anna-vasilko 🙏

[BryanA2T]

Doubling down on how important this is. We are unable to roll out to certain clients because of this issue.

[jeffss99]

Any update on this? It is super critical for us

[dmajor3]

Any update on this? Also can critical for us.

[manjeshbhargav]

We are working on this as our top priority, and we are aiming for a fix by end of this year. Please watch this space updates.

[manjeshbhargav]

Hi devs,

There is a new RC available with a fix for this issue: npm install twilio/twilio-video-processors.js#2.1.0-rc2 (script tag version bundle and assets available here). Please try it out and let us know if it helps. I will expedite the publishing of the production version 2.1.0 based on your feedback and our QA process.

[manjeshbhargav]

v2.1.0 is now available. Closing the issue.