ExoPlayer Custom Renderer

dhruvsingla273 commented 5 months ago

Hi All,

I am trying to create a player in which I want to do processing on frames of the video. So, I need to get each frame of the video, do some processing on it, and have to then render it to the Screen. I searched but couldn't find a compelling method that can be fast enough to render.

Can someone guide me for this how to make the custom renderer.

droid-girl commented 5 months ago

Hi @dhruvsingla273, You can use .setVideoEffects in ExoPlayer to achieve this:

Create a custom effect by extending BaseGlShaderProgram. You can find a few examples in the effects module, for example, ColorLutShaderProgram
Do frame modifications in drawFrame method in your custom shader program
in ExoPlayer, use setVideoEffects and set your custom effect.

I think this should help you to achieve your goal.

dhruvsingla273 commented 5 months ago

Hi @droid-girl , Thanks for the reply. Sorry for not being clear about the processing part in the last question. Actually I need to apply a TFlite model to each of the frames and I think shaders can't do that. I tried using MediaCodec to custom extract the frames and then apply the model and render them (https://github.com/duckyngo/Fast-Video-Frame-Extraction). It worked but I need to do it within ExoPlayer.

I tried making a custom Renderers Factory from MediaCodecVideoRenderer, but the ByteBuffer I got was empty as I got to know that Media Codec runs in Surface mode and hence it is not storing anything in buffers to improve codec performance. I could try to set MediaCodec to byteBuffer mode to get the byteBuffer but I read it is very slow. (Also can you tell how to set this mode change for MediaCodec in ExoPlayer)

Could you suggest a different approach?

dhruvsingla273 commented 5 months ago

I have one more question. If I can get a steady stream of a Bitmaps or ByteBuffer with my final frame data, can I somehow use ExoPlayer to render them using the player? Means I won't be having all the frames at a single time, I can have like say 2 sec of frames and then continue processing other frames and keep passing to ExoPlayer to render.

I know it might be confusing, please ask any clarification on the question you may need. Thanks again

dhruvsingla273 commented 5 months ago

Hi @droid-girl Have you found any way to integrate TFlite model?

droid-girl commented 5 months ago

Hi @dhruvsingla273 , Let me first provide a short summary on the approach you can take to integrate with TFLite:

Extend [BaseGlShaderProgram] as recommended in the earlier comment (https://github.com/androidx/media/blob/release/libraries/effect/src/main/java/androidx/media3/effect/BaseGlShaderProgram.java)
in drawFrame method you want to read frame to be able to pass it through TFLite:

    ByteBuffer pixelBuffer = ByteBuffer.allocateDirect(width * height * 4);
    Bitmap bitmap;
    int texId;
    try {
      int[] boundFramebuffer = new int[1];
      GLES20.glGetIntegerv(GLES20.GL_FRAMEBUFFER_BINDING, boundFramebuffer, /* offset= */ 0);

      int fboId = GlUtil.createFboForTexture(inputTexId);
      GlUtil.focusFramebufferUsingCurrentContext(fboId, width, height);
      GLES20.glReadPixels(
          /* x= */ 0,
          /* y= */ 0,
          width,
          height,
          GLES20.GL_RGBA,
          GLES20.GL_UNSIGNED_BYTE,
          pixelBuffer);
      GlUtil.checkGlError();
      bitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888);
      bitmap.copyPixelsFromBuffer(pixelBuffer);

Process bitmap in TFLite
Pass the processed output image

      texId =
          GlUtil.createTexture(
              outputBitmap.getWidth(),
              outputBitmap.getHeight(),
              /* useHighPrecisionColorComponents= */ false);
      GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, texId);
      GLES20.glTexParameterf(GL10.GL_TEXTURE_2D, GL10.GL_TEXTURE_MIN_FILTER, GL10.GL_NEAREST);
      GLES20.glTexParameterf(GL10.GL_TEXTURE_2D, GL10.GL_TEXTURE_MAG_FILTER, GL10.GL_LINEAR);
      GLES20.glTexParameterf(GL10.GL_TEXTURE_2D, GL10.GL_TEXTURE_WRAP_S, GL10.GL_REPEAT);
      GLES20.glTexParameterf(GL10.GL_TEXTURE_2D, GL10.GL_TEXTURE_WRAP_T, GL10.GL_REPEAT);
      GLUtils.texImage2D(GLES20.GL_TEXTURE_2D, /* level= */ 0, outputBitmap, /* border= */ 0);

Pass it to your glProgram glProgram.setSamplerTexIdUniform("uTexSampler", texId, /* texUnitIndex= */ 0);
Use setVideoEffects in ExoPlayer to set your custom effect

Here is just a high level overview of what needs to be done

dhruvsingla273 commented 5 months ago

Thanks for the response I will try this approach and let you know.

Meanwhile If I can get a steady stream of a Bitmaps or ByteBuffer with my final frame data (using a custom decoder from MediaCodec), can I somehow use ExoPlayer to render them using the player? Means I won't be having all the frames at a single time, I can have like say 2 sec of frames and then continue processing other frames and keep passing to ExoPlayer to render.

Do you have idea how can we do this?

droid-girl commented 5 months ago

@tonihei could you help with the last question?

tonihei commented 5 months ago

That's not possible at the moment because there is no mode in which you can render existing decoded frames from ByteBuffers. ExoPlayer supports image playback, as a series of Bitmaps, but this sounds very inefficient as you have to move the fully decoded buffers around. I think it's usually preferable to do all processing on the GPU? I think that's the approach @droid-girl explained above.