search

google / ExoPlayer

This project is deprecated and stale. The latest ExoPlayer code is available in https://github.com/androidx/media
https://developer.android.com/media/media3/exoplayer
Apache License 2.0
21.74k stars 6.03k forks source link

ABR DASH randomly changes video size within single track every ~10 seconds #5645

Closed oleksandr-yefremov closed 5 years ago

oleksandr-yefremov commented 5 years ago

[REQUIRED] Searched documentation and issues

Looked through samples, javadocs for the mentioned methods, issues on github (opened and closed) and dev guide.

[REQUIRED] Question

Situation: when playing DASH livestreams and VoDs quality changes very frequently (every 6-20 seconds, network is Wifi around 100Mbps, bandwidth is stable). According to this comment and my previous observations onDecoderInputFormatChanged should be called whenever track changes adaptively (i.e. if there are multiple tracks within the manifest, the current track switches to accomodate the bandwidth). What we see is there is one onDecoderInputFormatChanged callback and many onVideoSizeChanged with the resolution of one of the available tracks. I put our DRM'ed livestream into ExoPlayer demo app to exclude any differences in setting up the player. I see the same behaviour even if I disable adaptivity by passing params to DefaultTrackSelector.ParametersBuilder().setForceHighestSupportedBitrate(true).build(); in code or from ExoPlayer demo app (go to "Video" menu and select any single track manully).

Short version of log. Longer version is attached as file.

ExoPlayer 2.9.4.

2019-03-15 16:32:11.984 1700-1700/com.google.android.exoplayer2.demo D/EventLogger: decoderEnabled [0.07, 851.94, window=0, period=0, video]
2019-03-15 16:32:11.985 1700-1700/com.google.android.exoplayer2.demo D/EventLogger:       [ ] Track:0, id=livestream_video_01/cenc, mimeType=video/avc, bitrate=266240, codecs=avc1.42801F, res=416x234, fps=25.0, supported=YES
2019-03-15 16:32:11.985 1700-1700/com.google.android.exoplayer2.demo D/EventLogger:       [ ] Track:1, id=livestream_video_03/cenc, mimeType=video/avc, bitrate=1392640, codecs=avc1.42801F, res=640x360, fps=25.0, supported=YES
2019-03-15 16:32:11.985 1700-1700/com.google.android.exoplayer2.demo D/EventLogger:       [ ] Track:2, id=livestream_video_02/cenc, mimeType=video/avc, bitrate=718848, codecs=avc1.42801F, res=640x360, fps=25.0, supported=YES
2019-03-15 16:32:11.985 1700-1700/com.google.android.exoplayer2.demo D/EventLogger:       [X] Track:3, id=livestream_video_04/cenc, mimeType=video/avc, bitrate=2140160, codecs=avc1.4D4020, res=768x432, fps=25.0, supported=YES
2019-03-15 16:32:12.000 1700-1700/com.google.android.exoplayer2.demo D/EventLogger: downstreamFormatChanged [0.09, 851.94, window=0, period=0, id=livestream_video_04/cenc, mimeType=video/avc, bitrate=2140160, codecs=avc1.4D4020, res=768x432, fps=25.0]
2019-03-15 16:32:12.310 1700-1700/com.google.android.exoplayer2.demo D/EventLogger: decoderInitialized [0.40, 851.94, window=0, period=0, video, OMX.Exynos.avc.dec.secure]
2019-03-15 16:32:12.310 1700-1700/com.google.android.exoplayer2.demo D/EventLogger: decoderInputFormatChanged [0.40, 851.94, window=0, period=0, video, id=livestream_video_04/cenc, mimeType=video/avc, bitrate=2140160, codecs=avc1.4D4020, res=768x432, fps=25.0]
2019-03-15 16:32:12.581 1700-3100/com.google.android.exoplayer2.demo D/debug: pendingFormat : Format(livestream_video_04/cenc, null, null, video/avc, avc1.4D4020, 2140160, null, [768, 432, 25.0], [-1, -1])
2019-03-15 16:32:12.719 1700-3100/com.google.android.exoplayer2.demo D/debug: onOutputFormatChanged : {crop-right=767, vendor.rtc-ext-dec-low-latency.enable=0, color-format=291, slice-height=432, mime=video/raw, stride=768, color-standard=2, color-transfer=5, crop-bottom=431, vendor.rtc-ext-enc-caps-vt-driver-version.number=180315, crop-left=0, width=768, color-range=2, crop-top=0, height=432}
2019-03-15 16:32:12.721 1700-1700/com.google.android.exoplayer2.demo D/EventLogger: videoSizeChanged [0.81, 851.94, window=0, period=0, 768, 432]
2019-03-15 16:32:42.954 1700-3100/com.google.android.exoplayer2.demo D/debug: onOutputFormatChanged : {crop-right=639, vendor.rtc-ext-dec-low-latency.enable=0, color-format=291, slice-height=368, mime=video/raw, stride=640, color-standard=2, color-transfer=5, crop-bottom=359, vendor.rtc-ext-enc-caps-vt-driver-version.number=180315, crop-left=0, width=640, color-range=2, crop-top=0, height=368}
2019-03-15 16:32:42.956 1700-1700/com.google.android.exoplayer2.demo D/EventLogger: videoSizeChanged [31.04, 852.03, window=0, period=0, 640, 360]
2019-03-15 16:33:12.843 1700-3100/com.google.android.exoplayer2.demo D/debug: onOutputFormatChanged : {crop-right=767, vendor.rtc-ext-dec-low-latency.enable=0, color-format=291, slice-height=432, mime=video/raw, stride=768, color-standard=2, color-transfer=5, crop-bottom=431, vendor.rtc-ext-enc-caps-vt-driver-version.number=180315, crop-left=0, width=768, color-range=2, crop-top=0, height=432}
2019-03-15 16:33:12.963 1700-1700/com.google.android.exoplayer2.demo D/EventLogger: videoSizeChanged [61.05, 851.96, window=0, period=0, 768, 432]
2019-03-15 16:33:18.957 1700-3100/com.google.android.exoplayer2.demo D/debug: onOutputFormatChanged : {crop-right=415, vendor.rtc-ext-dec-low-latency.enable=0, color-format=291, slice-height=240, mime=video/raw, stride=416, color-standard=2, color-transfer=5, crop-bottom=233, vendor.rtc-ext-enc-caps-vt-driver-version.number=180315, crop-left=0, width=416, color-range=2, crop-top=0, height=240}
2019-03-15 16:33:18.961 1700-1700/com.google.android.exoplayer2.demo D/EventLogger: videoSizeChanged [67.05, 846.05, window=0, period=0, 416, 234]

outputIndex - ExoPlayer 2.9.4.txt videoSizeChanged - ExoPlayer 2.5.0.txt videoSizeChanged - ExoPlayer 2.9.4.txt

Questions:

  1. My understanding was decoderInputFormatChanged and videoSizeChanged should be pairwise, except for the cases when decoder remains the same and video surface size changes (e.g. rotated). Is that correct?
  2. I traced videoSizeChanged callback to drainOutputBuffer and outputIndex == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED. Tried looking further into Android's MediaCodec, but did not understand exactly what could be the reason for output format to change within a single track. Does this mean the actual segments are different? That sounds improbable because same DASH source is played on any desktop player without issues. Where can I start digging more?

Appreciate any hints or help.

tonihei commented 5 years ago

My understanding was decoderInputFormatChanged and videoSizeChanged should be pairwise, except for the cases when decoder remains the same and video surface size changes (e.g. rotated). Is that correct?

That's mostly the case, but not necessarily. For your case it looks as if the stream with id=livestream_video_04/cenc actually changes its size within the stream itself. That is, the decoded output samples are having different sizes. So even though the declared input format is still the same and ExoPlayer's track selection didn't change its decision, the output video size may change. If that is indeed the issue, it's probably an encoding issue that needs to be solved on the server.

That sounds improbable because same DASH source is played on any desktop player without issues. Where can I start digging more?

If at all possible, please provide us with an example stream sent to dev.exoplayer@gmail.com so that we can take a look.

oleksandr-yefremov commented 5 years ago

Ok, thanks for quick reply! I will check with our backend team to find out more about actual encoding. If we won't find anything suspicious or we prove that other players handle same input differently, I will try to prepare a sample livestream (probably with a temporary token) and send it to the email above. Thanks again!

tonihei commented 5 years ago

Thanks for the provided information. I was able to reproduce the problem and can see what is happening now.

The manifest updates change the order of the representations in the adaptation set. ExoPlayer assumes that the representations stay in the same order which is also what the DASH-IF interoperability guidelines will suggest in the next version (see here). We keep playing whatever is the i th position in the list, that's why the changing formats.

One example for a manifest update I've seen in the test stream:

<AdaptationSet group="2" contentType="video" /* ... */ >
<Representation id="video_04/cenc" bandwidth="3891200" width="1280" height="720" codecs="avc1.4D4020" scanType="progressive"/>
<Representation id="video_05/cenc" bandwidth="4710400" width="1920" height="1080" codecs="avc1.4D402A" scanType="progressive"/>
<Representation id="video_02/cenc" bandwidth="1331200" width="768" height="432" codecs="avc1.4D4020" scanType="progressive"/>
<Representation id="video_01/cenc" bandwidth="573440" width="640" height="360" codecs="avc1.4D401F" scanType="progressive"/>
<Representation id="video_03/cenc" bandwidth="2826240" width="1024" height="576" codecs="avc1.4D4020" scanType="progressive"/>
</AdaptationSet>
<AdaptationSet group="2" contentType="video" /* ... */ >
<Representation id="video_02/cenc" bandwidth="1331200" width="768" height="432" codecs="avc1.4D4020" scanType="progressive"/>
<Representation id="video_01/cenc" bandwidth="573440" width="640" height="360" codecs="avc1.4D401F" scanType="progressive"/>
<Representation id="video_04/cenc" bandwidth="3891200" width="1280" height="720" codecs="avc1.4D4020" scanType="progressive"/>
<Representation id="video_03/cenc" bandwidth="2826240" width="1024" height="576" codecs="avc1.4D4020" scanType="progressive"/>
<Representation id="video_05/cenc" bandwidth="4710400" width="1920" height="1080" codecs="avc1.4D402A" scanType="progressive"/>
</AdaptationSet>

Would it be possible to ensure that the order of representations stays the same for all updates?

ojw28 commented 5 years ago

which is also what the DASH-IF interoperability guidelines will suggest

IIRC it's going to be mandatory for DASH-IF interoperability, not just a suggestion.

oleksandr-yefremov commented 5 years ago

Ahh, now it all makes sense! I had a feeling that it could be related to manifest updates, but did not check whether current track is determined by index or by id. Looks like Bitmovin and Shaka use id and that's why there is no issue. Yes, we will make sure that order of tracks is maintained across updates, especially if that will be mandatory in future. Thank you for support! I'm closing the ticket.