video-processor pipeline WebGL2 throws TypeError: can't access property "VERTEX_SHADER", gl is null

[thlee1122]

I found a bug with WebGL2. We recently upgraded the twilio-video-processor to the latest version, 2.0.0. With the above-updated version, the WebGL2 pipeline video background blur functionality throws a TypeError: can't access property "VERTEX_SHADER", gl is null.

Below is the screenshot of the error:

Software versions:

• [ ] Browser(s): Chrome (Version 114.0.5735.106)
• [ ] Operating System: MacOS Ventura 13.4
• [ ] twilio-video-processors.js: 2.0.0
• [ ] twilio-video: 2.27.0
• [ ] Third-party libraries (e.g., Angular, React, etc.): React

[alex93293]

I managed to find the root cause for this.

The default pipeline for the video processors is set to WebGL2, but the default processor options in twilio-video is set to { inputFrameBufferType: 'offscreencanvas', outputFrameBufferContextType: '2d' }, which are options for Canvas2D.

There are 2 ways to fix this:

1. Pass WebGL2 options to track.addProcessor

   track.addProcessor(processor, { inputFrameBufferType: "video", outputFrameBufferContextType: "webgl2" });

2. Use Canvas2D pipeline*

   processor = new VirtualBackgroundProcessor({
   pipeline: Pipeline.Canvas2D,
   ...options
   });

I only managed to find this out by reading through the example.

*Canvas2D might not have full functionality with Safari - the Gaussian Blur didn't work for Safari 16.5.1 MacOS (haven't tested others)

[thlee1122]

@alex93293 Thank you for this! This looks like a work around. Is the Twilio video processor team going to release an actual fix for this issue?

[lgenzelis]

Also worked for me! Does anyone knows which one should we be using? (webgl2 or canvas2d) I mean, since the defaults are different for @twilio/video-processors and twilio-video.

As a sidenote, I'm sticking with canvas2d for now, since webgl2 completely ignores the fitType option (and having a ImageFit.Contain is must for me).

[sajitkhadka]

I had the same issue when I upgraded to v2.0.0. The following seemed to work for me but it was painfully slow. so I ended up using Canvas2D pipeline option instead. track.addProcessor(processor, { inputFrameBufferType: "video", outputFrameBufferContextType: "webgl2" });

Edit: It seems for higher resolution it is slow. width: 1280, height: 720, if I change width:640, height:480. it seems to be okay.