Support splicing in DASH and SS to better support adaptive quality changes after load cancellation in the middle of the segment

[RamilGabdrakhmanov]

ExoPlayer can change video track during DASH adaptive playback. How i know ExoPlayer can change video track only on segments edges. So, can ExoPlayer change video track the middle of the segment?

Example: 1) Each segment is 10 seconds long 2) 4 second of the segment was played 3) I have only 0,5 sec buffer 4) Bandwidth is very low 5) I recognize that it is necessary to change video track to lower quality, to avoid stalled Current behavior: Player may fall into stalled, will play segment to the end, next one will be downloaded in lower quality

Desired behavior: Player cancels current segment download, starts to download current segment in lower quality, and without stalled will change quality in the middle of the segment

Is desired behavior possible?

[christosts]

The short answer is you should avoid doing that. It is highly unlikely that playback will remain smooth and (my understanding is that) changing quality mid-segment goes against the adaptive specs design.

To change to a different segment while playing the same segment in a different quality, we'd have to start downloading the new segment from its start and, for video segments, we'd have to start decoding from the nearest key-frame before the position we need to start presenting frames. Usually, that key-frame is the first frame in the segment, hence it's most likely that the device will need to decode (but not display) all preceding frames in the segment. If you decide to switch a 10-second segment after you played half of it, you'd need to decode the preceding 5 seconds of the lower quality segment. It is more likely that this will not happen fast enough to ensure smooth playback.

To avoid rebuffering, it think it's best to tune the parameters that are expected to be tuned by design. On the player side, change the adaptation algorithm and (if you think it's needed), the bandwidth meter mechanism. For example, design your algorithm to always ensure there is enough data buffered (more that 1 segment?) On the overall system design, if you control encoding/packaging, use smaller segments (2-5 seconds) if you want to adapt to vast changing network conditions.

Whether is it possible: @tonihei after merging #7244, is it possible for an app to stop downloading a currently playing HLS segment? And for DASH, the answer is no, right?

[RamilGabdrakhmanov]

We are developing low latency live dash playback. Our target latency is 7 seconds. So, usually we will have 3-4 sec buffer. 3-4 sec will spend to transcode video and deliver it to CDNs. And we have 4-second segments.

In stable network situation we will change video quality just between segments. But sometimes network can get worse very quickly. In this cases much better to re-download and decode same segment but in lower quality, then get stalled

[tonihei]

As @christosts pointed out, the preferred way for adaptive quality adjustment is to make changes at segment boundaries as intended by the specs. However, cancelling an ongoing load if the chosen quality was too high is a valid use case we want to support as well.

ExoPlayer already supports canceling an ongoing chunk load to allow reloading it in another quality. This was implemented as part of #7244 and works for DASH, HLS and SmoothStreaming. It isn't enabled by default though and you need to implement your own logic to decide when this should happen by subclassing AdaptiveTrackSelection or writing a fork of this class with a custom implementation in shouldCancelChunkLoad.

However, cancelling a chunk that the decoder already read from requires to splice-in the data from the newly loaded chunk to get a transition in the middle of the segment. This splicing operation is only supported in HLS at the moment. So while you can cancel ongoing loads for all adaptive streaming formats, the splicing works only for HLS. AdaptiveTrackSelection.shouldCancelChunkLoad is therefore not called when the cancellable chunk load is for a chunk a decoder has already read from in DASH and SmoothStreaming (see code).

I'll mark this as an enhancement to support splicing for DASH and SmoothStreaming. It's unlikely we start on work on this soon though.