ExoPlayer Custom Renderer

[dhruvsingla273]

Hi All,

I am trying to create a player in which I want to do processing on frames of the video. So, I need to get each frame of the video, do some processing on it, and have to then render it to the Screen. I searched but couldn't find a compelling method that can be fast enough to render.

Can someone guide me for this how to make the custom renderer.

[droid-girl]

Hi @dhruvsingla273, You can use .setVideoEffects in ExoPlayer to achieve this:

1. Create a custom effect by extending BaseGlShaderProgram. You can find a few examples in the effects module, for example, ColorLutShaderProgram
2. Do frame modifications in drawFrame method in your custom shader program
3. in ExoPlayer, use setVideoEffects and set your custom effect.

I think this should help you to achieve your goal.

[dhruvsingla273]

Hi @droid-girl , Thanks for the reply. Sorry for not being clear about the processing part in the last question. Actually I need to apply a TFlite model to each of the frames and I think shaders can't do that. I tried using MediaCodec to custom extract the frames and then apply the model and render them (https://github.com/duckyngo/Fast-Video-Frame-Extraction). It worked but I need to do it within ExoPlayer.

I tried making a custom Renderers Factory from MediaCodecVideoRenderer, but the ByteBuffer I got was empty as I got to know that Media Codec runs in Surface mode and hence it is not storing anything in buffers to improve codec performance. I could try to set MediaCodec to byteBuffer mode to get the byteBuffer but I read it is very slow. (Also can you tell how to set this mode change for MediaCodec in ExoPlayer)

Could you suggest a different approach?

[dhruvsingla273]

I have one more question. If I can get a steady stream of a Bitmaps or ByteBuffer with my final frame data, can I somehow use ExoPlayer to render them using the player? Means I won't be having all the frames at a single time, I can have like say 2 sec of frames and then continue processing other frames and keep passing to ExoPlayer to render.

I know it might be confusing, please ask any clarification on the question you may need. Thanks again

[dhruvsingla273]

Hi @droid-girl Have you found any way to integrate TFlite model?

[droid-girl]

Hi @dhruvsingla273 , Let me first provide a short summary on the approach you can take to integrate with TFLite:

1. Extend [BaseGlShaderProgram] as recommended in the earlier comment (https://github.com/androidx/media/blob/release/libraries/effect/src/main/java/androidx/media3/effect/BaseGlShaderProgram.java)
2. in drawFrame method you want to read frame to be able to pass it through TFLite:

    ByteBuffer pixelBuffer = ByteBuffer.allocateDirect(width * height * 4);
    Bitmap bitmap;
    int texId;
    try {
      int[] boundFramebuffer = new int[1];
      GLES20.glGetIntegerv(GLES20.GL_FRAMEBUFFER_BINDING, boundFramebuffer, /* offset= */ 0);

      int fboId = GlUtil.createFboForTexture(inputTexId);
      GlUtil.focusFramebufferUsingCurrentContext(fboId, width, height);
      GLES20.glReadPixels(
          /* x= */ 0,
          /* y= */ 0,
          width,
          height,
          GLES20.GL_RGBA,
          GLES20.GL_UNSIGNED_BYTE,
          pixelBuffer);
      GlUtil.checkGlError();
      bitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888);
      bitmap.copyPixelsFromBuffer(pixelBuffer);

3. Process bitmap in TFLite
4. Pass the processed output image

      texId =
          GlUtil.createTexture(
              outputBitmap.getWidth(),
              outputBitmap.getHeight(),
              /* useHighPrecisionColorComponents= */ false);
      GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, texId);
      GLES20.glTexParameterf(GL10.GL_TEXTURE_2D, GL10.GL_TEXTURE_MIN_FILTER, GL10.GL_NEAREST);
      GLES20.glTexParameterf(GL10.GL_TEXTURE_2D, GL10.GL_TEXTURE_MAG_FILTER, GL10.GL_LINEAR);
      GLES20.glTexParameterf(GL10.GL_TEXTURE_2D, GL10.GL_TEXTURE_WRAP_S, GL10.GL_REPEAT);
      GLES20.glTexParameterf(GL10.GL_TEXTURE_2D, GL10.GL_TEXTURE_WRAP_T, GL10.GL_REPEAT);
      GLUtils.texImage2D(GLES20.GL_TEXTURE_2D, /* level= */ 0, outputBitmap, /* border= */ 0);

5. Pass it to your glProgram glProgram.setSamplerTexIdUniform("uTexSampler", texId, /* texUnitIndex= */ 0);

6. Use setVideoEffects in ExoPlayer to set your custom effect

Here is just a high level overview of what needs to be done

[dhruvsingla273]

Thanks for the response I will try this approach and let you know.

Meanwhile If I can get a steady stream of a Bitmaps or ByteBuffer with my final frame data (using a custom decoder from MediaCodec), can I somehow use ExoPlayer to render them using the player? Means I won't be having all the frames at a single time, I can have like say 2 sec of frames and then continue processing other frames and keep passing to ExoPlayer to render.

Do you have idea how can we do this?

[droid-girl]

@tonihei could you help with the last question?

[tonihei]

That's not possible at the moment because there is no mode in which you can render existing decoded frames from ByteBuffers. ExoPlayer supports image playback, as a series of Bitmaps, but this sounds very inefficient as you have to move the fully decoded buffers around. I think it's usually preferable to do all processing on the GPU? I think that's the approach @droid-girl explained above.

[dhruvsingla273]

Hi @droid-girl, using the effect method I was able to get the bitmap at each frame, But I am not able to render anything on the surface. So my surface is coming as black. I even tried just passing the as it is bitmap without any change to the surface but it is not rendering.

Any idea why?

[droid-girl]

It is hard to say without looking at the code and debugging it. Please check if you have any GL errors and if you correctly bind processed frames