Video freezes and audio continues to play on ABR switch, reproducible on Skyworth Android(10) STB, Amlogic S905X2

[luckygoyal-bitmovin]

ExoPlayer Version

2.16.1

Devices that reproduce the issue

Skyworth Digital Android STB running Android 10 and using Amlogic S905X2 chipset

Devices that do not reproduce the issue

Issue did not occur on tested Android Mobile devices(Samsung S21, OnePlus AC 2003) etc.

Reproducible in the demo app?

Yes

Reproduction steps

• Play Widevine protected + HEVC + AAC live DASH media in Demo application (Can provided media details over email)
• Let playback happen for some time. Please ensure that ABR switching happens during playback

Expected result

• Video freeze should not happen on ABR switch

Actual result

• Video freezes and audio continues to play after one of the ABR switches. The freeze is intermittent and does not occur on every ABR switch. The same to->from rendition switch combination work fine sometimes but leads to video freeze at other times.
• There is no error or exception coming from MediaCodec.
• Player continues to download audio and video segments and audio only playback continues
• Video back buffer continues to grow over time and leads to playback failure eventually.

Media

Will send media details to dev.exoplayer@gmail.com

Bug Report

• [X] You will email the zip file produced by adb bugreport to dev.exoplayer@gmail.com after filing this issue.

[luckygoyal-bitmovin]

Hi Exoplayer team, I have sent the bugreport and stream information over email. The issue is also reproducible on latest Exoplayer release-v2 branch. The issue is also reproducible with clear stream decoded using non-secure decoder OMX.amlogic.hevc.decoder.awesome2 instead of OMX.amlogic.hevc.decoder.awesome2.secure which is used for encrypted content.

I understand it may not be possible to reproduce the issue if you do not have the specific STB Model (or any Amlogic S905X2 STB device), in which case any debug steps to isolate the problem to player/decoder/stream will be really helpful.

[luckygoyal-bitmovin]

Hi Exo team, any feedback on this ticket please?

[gb103]

Any resolution on this, am also facing the same issue. My exoplayer version is 2.15.1

[christosts]

This sounds like a device-specific problem but let's try to strike out some other possibilities first.

On the email we received, the test content is protected with Widevine and the bug-report has some CDM errors logged by the platform as soon as playback starts (not after an adaptive switch). You also mentioned that the issue reproduces with clear content, so it's not related to DRM. Can you please send us a link to the clear content and a bug-report after the issue reproduces with the clear content?

You also mentioned that the issue reproduces with 2.16.1 and the latest ExoPlayer release-v2 branch (I assume that's 2.18.0?). Did this content ever work on this device with an ExoPlayer version before 2.16.1?

You also mention

Video back buffer continues to grow over time and leads to playback failure eventually.

I did not see a related error on the bug-report. Did you see a player error reported by the player about this?

[luckygoyal-bitmovin]

Hi @christosts , thanks for checking this issue. We have made some progress on analysis of this issue since reporting it and have a hypothesis. I am sharing it below to get a feedback and hopefully any further direction from Exo team.

• This indeed seems like a device specific issue but is not specific to device but rather a combination of device and stream.
• With further tests we identified that issue occurs on ABR switching between specific pairs of video renditions.
• Further analyzing the affected to and from renditions leading to the reported behaviour, we found that the HEVC encoding of these renditions use a different number of reference_frame configuration. This information can be found by analyzing the VPS/SPS units in init segments which are in clear(pls see screenshot below for reference).
  • 288p, HEVC Main, Level 3.1, 350Kbps, 15 reference frames
  • 432p, HEVC Main, Level 3.1, 800Kbps, 11 reference frames
  • 540p, HEVC Main, Level 3.1, 1200Kbps, 7 reference frames
  • 720p, HEVC Main, Level 4.1, 2000Kbps, 11 reference frames
  • 1080p, HEVC Main, Level 4.1, 3500Kbps, 5 reference frames

• If playback starts from 540p rendition(using 5 ref_frames) then switching down to 288p renditions(using 15 ref_frames) causes freeze to occur 100% of the times. Similarly starting playback from rendition with lower ref_frames config and then switching to rendition using higher ref_frames causes freeze to occur with varying degree of repro rate.

• Another observation was that issue did not occur if playback started with lowest rendition 288p which uses maximum number of ref_frames(15).

• Based on above observations, we arrived at a hypothesis that the issue could be related to decoder framework's handling of ref_frames config. at the start of playback and it's adaptability as the ref_frames requirements change after ABR switch. To strengthen this hypothesis further we added additional logs to MediaDecoderRenderer and MediaVideoDecoderRenderer in Exoplayer and found that when the issue occurs, the MediaVideoDecoderRenderer in Exoplayer is not getting decoded output buffers corresponding to the switched to rendition (288p, 350 Kbps HEVC). The sequence of decoding and rendering in player works as follows.

  • Player downloads and parses the video frames and enqueues these into media decoder for decoding
  • Then player periodically polls the decoder for processed output frames so that it can render and release the out buffer used for storing decoded frame.
  • When ABR switch happens, for seamless switching the player prefixes the codec configuration information of the new video rendition before first frame of the new vide rendition while enqueuing it for decoding. This enables decoder to know that the next frame(s) to decode belong to a new rendition so that decoder can adjust accordingly.
  • In working case, that is when ABR switch to lowest rendition is working as expected, all above steps work as expected and player is able to continuously enqueue video frames and process output buffers successfully.
  • But in video freeze case, the decoder stops providing decoded output buffers for processing starting from the first video frame of switched to rendition.

We are now checking this with STB provider. But would be really helpful to have your feedback on same.

• Does above hypothesis seem to be in the right direction?
• Is there anything that can be done in Exoplayer to bypass/workaround this problem? For example any possibility to initialize the decoder with maximum number of reference frames configured in ABR ladder.
• In your experience, is having different ref_frames configuration for different video renditions a common practice?

[christosts]

It sounds like this is a device-specific problem.

Does above hypothesis seem to be in the right direction?

Yes. If a MediaCodec instance stops producing input/output buffers, the player will get stuck. This is aligned with your observation of video freeze (the last decoded frame remains on screen). Unfortunately, we do not detect such cases as it is very difficult to reliably detect a decoder is stuck.

Is there anything that can be done in Exoplayer to bypass/workaround this problem? For example any possibility to initialize the decoder with maximum number of reference frames configured in ABR ladder.

The renderer expects the initialization data to be in the Format.initializationData. For the renderer, this is just a byte-sequence. I guess it's better/easier to override what the renderer does rather than how the extractor reads the data from the container. You could try to override MediaCodecVideoRenderer.onInputFormatChanged() (see code) and tweak the format's initialization data (or copy the format instance) before calling super.onInputFormatChanged(formatHolder). I have not tried this though, so I cannot guarantee it will work.

In your experience, is having different ref_frames configuration for different video renditions a common practice?

I don't have something in my mind, but this is beyond my expertise.

Since this is a device-specific problem, I'll close this issue, Feel free to re-open if there's ExoPlayer issue that you come across with.