Reuse video texture for mask mixing

[danrossi]

MediaPipe Solution (you are using)

selfie segmentation

Programming language

Javascript

Are you willing to contribute it

None

Describe the feature and the current behaviour/state

Ability to apply a video texture from the same webgl context instead of reuploading a video texture. Like applying the mask texture.

Will this change the current API? How?

No response

Who will benefit with this feature?

No response

Please specify the use cases for this feature

WebGL mixing of video, mask and background image

Any Other info

I currently use this in my render to mix elements with a custom shader. However it might be more efficient to reapply the already added video texture in media pipe than reuploading a video frame. I already apply the mask texture from the segmentation result. And an image texture is already added to the program and doesnt need to be reuploaded. It's video that is being reuploaded a second time after being sent to mediapipe.

Ideally a premixed output using the calculator already available to do such things. Doesn't require post processing in webgl.

 gl.clearColor(1.0, 1.0, 1.0, 1.0);
        gl.useProgram(this.prog);
        gl.clear( gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT );
        gl.viewport(0, 0, video.videoWidth, video.videoHeight);

        const texture = results.confidenceMasks[0].getAsWebGLTexture();

        this.bindBuffers(gl, this.positionLocation, this.texCoordLocation);

        gl.activeTexture(gl.TEXTURE0);
        gl.bindTexture(gl.TEXTURE_2D, this.bgTexture);

        gl.activeTexture(gl.TEXTURE2);
        gl.bindTexture(gl.TEXTURE_2D, texture);
        gl.uniform1i(this.uniformLocations.mask, 2);

        gl.activeTexture(gl.TEXTURE1);
        gl.bindTexture(gl.TEXTURE_2D, this.videoTexture);
        gl.uniform1i(this.uniformLocations.frame, 1);

        gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGB, gl.RGB, gl.UNSIGNED_BYTE, video);
        gl.drawArrays(gl.TRIANGLE_FAN, 0, 4);

So ability to bind the already added video texture in the gl context of mediapipe like it's done with the mask

gl.bindTexture(gl.TEXTURE_2D, texture);

The processing is done on the same offscreen canvas context as mediapipe is using.

[kuaashish]

Hi @danrossi,

Could you kindly provide the detailed steps you are following from our documentation, or alternatively, share the standalone code with us? This will help us to reproduce the issue and gain a better understanding of it.

Thank you!!

[danrossi]

I'll find a vanilla code. Its actually the webgl post render on the same context. Hence binding the mask texture. Would be good somehow to apply the video texture the same to not have to reupload a video frame. It's already efficient enough but trying to improve it further.

[kuaashish]

Hi @danrossi,

We apologize for any confusion. Could you please confirm whether you are utilizing our new Task API as outlined in the documentation, or if you are still using our legacy Selfi Segmentation from here? Please note that support for legacy solutions has been discontinued, and they are no longer maintained. Our current focus is on enhancing our new Task APIs, so new features will only be added to those.

Thank you!!

[danrossi]

I was tied up with massive VR upgrades to fix issues with my Safari hacks and apple vision I am sorry. I have put up a working test file. requesting media device isnt possible in jsfiddle.

You can see this bit I have to reupload the video frame even from within the same context. Hopefully rebinding the internal video texture on the offscreen canvas context from mediapipe is enough to render video to be mixed ? Im assuming its scaled down so can be rescaled ?

rendering in webgl is performing ok enough Ive done performance tests. Just that one last bit I am hoping for. Like applying the mask texture from mediapipe.

gl.activeTexture(gl.TEXTURE1)
    gl.bindTexture(gl.TEXTURE_2D, videoTexture)
    gl.uniform1i(frameTextureLocation, 1)

    gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, video);

https://electroteque.org/dev/mediapipe/

Example it working although I bundle it in, so haven't made an update for a while. I'll double check.

https://electroteque.org/plugins/videojs/rtcstreaming/demos/virtual-background/

[kuaashish]

Hi @danrossi,

I apologize for the delayed response. Could you please confirm if this issue has been resolved on your end, or if you still require assistance from us?

Thank you!!

[danrossi]

It's a feature request. If I can grab the video texture already available like the mask texture it might be helpful. Is there a video texture created for the input source ? I apply a mix on the same offscreen canvas context used for mediapipe. Then copy that to the display canvas. Right now I have to reupload the video frame to another texture for mixing.

[danrossi]

snippet of the shader used to mix. It requires applying smoothing as the selfie segmentation model has no smoothing. Where the deeplab model does but it's less accurate of the edges and objects appearing in the background it picks up I found. So the frame texture I am hoping to obtain the one used within mediapipe to not have to reupload another video frame.

I've created a PlayCanvas integration using the same modules I built for the WebRTC feature. https://electroteque.org/plugins/playcanvas/virtual-background/

void main(void) {

    vec4 maskTex = texture(mask, v_texCoord);
    vec4 frameTex = texture(frame, v_texCoord);
    vec4 bgTex = texture(background, v_texCoord);
    vec3 frameColor = frameTex.rgb;
    vec3 backgroundColor = bgTex.rgb;
    float maskVal = 0.0;
    float u_lightWrapping = 0.1;

    //vec2 u_coverage = vec2(0.6,0.85);
    vec2 u_coverage = vec2(0.1, 0.5);

    maskVal = maskSmoothing(frameColor);

    frameColor = lightWrapping(frameColor, backgroundColor, maskVal, u_coverage);

    //outColor = mix(bgTex, frameTex, 1.0);
    outColor = mix(bgTex, vec4(frameColor, 1.0), maskVal);
    //outColor = mix(bgTex, vec4(frameColor, 1.0), maskTex.a);
}