setVideoEffects can take effect immediately

[GavinChu]

[REQUIRED] Use case description

Before the player is prepared, setVideoEffect has been called to set an effect. When playback is paused, we call setVideoEffect again to update the effect chain:

• The player cannot update the current screen with new effects.
• The new effects can only be applied when the player resumes playback.

We hope that the current screen will take effect immediately after calling setVideoEffect.

I took a preliminary look at the cause of the problem:

1. After MediaCodec releases a frame, it cannot send the same frame to ExternalTextureManager again.
2. The frame in ExternalTextureManager is released after being processed by the next GlShaderProgram.
3. setVideoEffect will trigger ExternalTextureManager#signalEndOfCurrentInputStream and reconfigure the special effects chain.

Proposed solution

I can't think of a feasible solution for the time being. Do you have a plan or solution?

[claincly]

This is a known improvement and we have tracked it internally, but we don't have a timeline for it yet.

It's quite complexed to fix this, I had a prototype and here's roughly the steps to do it:

• Keep a replyable cache of unprocessed frames at the start of the processing pipeline, after ExternalTextureManager, via possibly extending FrameCacheGlShaderProgram
• The cache will have at least two frames, one releases frame one when frame two is sent for display
• When setting new video effects, replay the frame in the cache and have the frame pass through the effect pipeline

[GavinChu]

Thanks for your reply, your prototype is great.

I further analyzed and found some interesting problems:

1. Cache inside the special effects chain Every time a GlShaderProgram is added, a FrameConsumptionManager is connected, and the FrameConsumptionManager can cache frames, which is basically 1 when playing the preview. This will cause FinalShaderProgramWrapper to render only one frame, but ExternalTextureManager may have advanced many frames. In this way, when caching unprocessed frames at the beginning of the processing pipeline, more frames need to be cached, which may affect memory.

2. SurfaceTexture frames will be released As I said before, updating the special effects chain will trigger ExternalTextureManager#signalEndOfCurrentInputStream, which will execute removeAllSurfaceTextureFrames to clear all cached frames in SurfaceTexture, and cause ExternalTextureManager to never get these frames again.

[claincly]

• The use of FrameConsumptionManager is to chain together GlShaderPrograms, and the premise is the shader programs can run on any thread they choose, that's why we have this rather convoluted way of signaling frames being ready, etc. However, FrameConsumptionManager itself just keeps an reference of the frames coming in and going out, it does not make copies of frames (thus no extra memory used). OTOH, if you use a frame cache, the cache needs to copy the incoming textures, using more memory.
• You are right there's more handling when seeking, and it's one of the harder problems in this already hard problem :)

[nesyou01]

Same issue