Getting the decoded from ExoPlayer in advance

[dhruvsingla273]

Hi everyone, I need to access video frames before they are displayed on the screen . Specifically, I want to retrieve future frames from ExoPlayer's buffer during video playback.

What I've Tried:

1. Custom MediaCodecVideoRenderer: I created a custom MediaCodecVideoRenderer and set the surface to null to access the ByteBuffer. Then in the function processOutputBuffer I tried to get the buffer. However, this buffer only updates when frames are being displayed. Video Effects:
2. I attempted to use .setVideoEffect and accessed frames using glReadPixels, but this approach also provides frames only at the time of rendering.

My Requirement: ExoPlayer must maintains a forward buffer of decoded frames(not sure). I need to access frames from this buffer before they are rendered. I want to process these frames and then render the processed frames using player.setVideoEffect.

Could anyone suggest how to achieve this or point me in the right direction?

[droid-girl]

Hi @dhruvsingla273 , Could you share what you are trying to do with those frames? Do you require an input of several frames to produce an effect or what is the main goal with this request?

[dhruvsingla273]

I want to upscale the frames using TFLite model. Refer this - https://github.com/androidx/media/issues/1422

So basically I want to preprocess some of the frames so that I have a buffer of upscaled frames which I can then render and keep processing other frames in background. So like say the player is paused at some point, then also exoplayer would be decoding the future frames to show, Now I want to get these future frames so that I can keep preprocessing the frames. i don't want to process the frames on the fly (like one frame comes, upscale it and then render it, rather I want to already have some upscaled frame ahead of rendering)

I will just pass these frames in a queue to my GL Program for rendering using .setVideoEffect (its a workaround of sorts)

Let me know if its still unclear. Thanks

[dhruvsingla273]

Hi @droid-girl Any updates?

Thanks

[kim-vde]

Hi,

If I understand your request correctly, you want to have the following structure:

decoder -> upscaling effect -> queue of pending frames -> surface

Is that correct?

If so, I think the FrameCache effect can help. You could pass a list of effects to ExoPlayer.setVideoEffects() that contains your upscaling effect, followed by a FrameCache effect.

decoder -> upscaling effect -> frame cache effect -> surface

[dhruvsingla273]

Hi, decoder -> upscaling effect -> frame cache effect -> surface

This is not the structure I want, I am looking for to get the frame in advance to process them (upscaling effect). If I am correctly understanding your method caches my upscaled frames. (let me know if I am wrong)

Rather I want to obtain raw frames earlier say 10 frames before I started rendering, then i can upscale these 10 frames in parallel and make a queue to render them.

[kim-vde]

With the frame cache, your upscaling effect should get the frames in advance.

If you don't add a frame cache between the upscaling effect and the surface, the upscaling effect will receive the frame right before it is rendered because there is no way to store a frame between the effect and the surface. The frames input to the effect are limited by back-pressure: no frame goes in if we can't handle the output.

If you have a cache of frames between the effect and the surface, then the effect will be able to receive frames earlier, because the output can be stored before it is rendered. Assuming decoding is faster than rendering, your frame cache should be full, and your effect should therefore receive frames in advance.

Does that make sense?

[realzzq]

hi, i have some reqirements that how to get the decoded picture frame data from the decoder. in ffmpeg is equals the AVFrame->data[0], data[1], data[2](YUV data), i wanna get these YuvData and then give other program to deal with. can U help me?Thanks @droid-girl

[kim-vde]

Hi,

This seems unrelated to the current issue. Could you open another issue or clarify why it's related?

Thanks

[realzzq]

ok