Overriding AdaptiveTrackSelection to adjust output of canSelectFormat for selective track playback based on resolution/bitrate

[Ste-RH]

Version 2.15.0

Apologies if this has been covered somewhere else...have been trawling through issues for a while and cannot find a solution or even a hint as to how to address the problem I am facing.

Essentially I am AdaptiveTrackSelection and it's factory so I can override 'canSelectFormat'. My aim is to limit what adaptive stream is downloaded/played on resolution and/or bitrate. I have gotten things to the point where I can chop back an HLS stream from initially playing 1080p to its lowest resolution, but then when I allow it to fly free again and let all tracks through again...it never climbs back up to 1080p.

I am testing with the apple bipbop stream and also tears of steel:

https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_16x9/bipbop_16x9_variant.m3u8 https://stream.mux.com/4XYzhPXzqArkFI8d1vDsScBLD69Gh1b2.m3u8

Hope someone can tell me what I am doing wrong! Thanks!

[christosts]

Please could you provide a minimal patch to the demo app so that we can reproduce this? Thanks!

[Ste-RH]

Sure thing! Patch made on latest pull of dev-v2.

I am using HLS->Apple 16x9 basic stream (TS) for testing.

If you filter logcat on 'STEEE' you will see when the limiting is changed first to 640x480 max, then up to 1920x1080. The switch back up never seems to happen, it gets stuck on playing back the low resolution limited stream.

Note that if I don't add the load controller with 5000ms buffering, the switch to a lower stream doesn't ever seem to happen (left it a handful of minutes). With the 5000ms buffering the change happens around 9 seconds into the video with the timings as things stand.

ExoPlayer_Issue9684_patch.zip

[christosts]

I see if your patch that:

1. You use custom values on the LoadControl, all params are set to 5 seconds. I'm assuming you want to queue less data so that resolution switching happens soon.
2. The first time you set the resolution limit on the VideoStreamLimiter is 5 seconds after the player is initialized. In my tests, the player had enough time in these 5 seconds to download a lot of data at 1080, so I tweaked the patch to set the 640x480 limit immediately.

I can verify that with the patch (tweaked) the player always remains at 640x480. The reason includes a few parameters which I will explain in more detail, but the short answer is: apart from canSelectFormat(), please consider overriding evaluateQueueSize() (more details in #7969) so that player may discard old chunks and start downloading high res chunks sooner. You should not have to change the load control values, at least not set max buffer that low. Please refer to #7969 for more information what the evaluateQueueSize() should do.

What happens in the patch and the player gets stuck at low res:

• The custom load control values limit the buffer duration to 5 seconds of data.
• In the specific HLS playlist you're testing, segment durations are ~10 seconds and segments are not independent.
• Although canSelectFormat() at some point does inform the player it can select the 1080 resolution, the adaptive bitrate algorithm will not attempt to switch.
• That's because for this specific playlist, with segments not being independent, for a resolution switch to happen at segment X, the player must download segments X-1 and X at the new resolution (segment X-1 must be downloaded in the new resolution too). These two segments occupy about 20 seconds of data, but the load control buffers up to 5 seconds, and the player's adaptive algorithm considers the change is too risky to cause a re-buffer (it needs to download data worth of 20 seconds with only 5 seconds buffered). So it does not change the resolution.

If you leave the load control values in their default setting, and do not override evaluateQueueSize(), you'll see the resolution changing at some point, possibly after 50 seconds if the queue is full. If you want to see the resolution change happen sooner, override evaluateQueueSize() to inform the player to discard some chunks and replace them with the new resolution. This is not very efficient because chunks are downloaded twice. You could also configure load control to queue less data than the default to mitigate the overhead, but set the max buffer to a higher value than 5 seconds (maybe 30 seconds?)

[Ste-RH]

Thanks for all the info, makes total sense and overriding evaluateQueueSize(...) is definitely the way to go I think in our use case.

So with that in mind I have reverted back to the default load controller buffering timings and overloaded and have written some code to control the number of chunks that are kept in the queue based upon time from current playhead. However, this is not doing what I expect. So, I have opted to simply return 0 from the overloaded function to check if the stream switches back up to 1920x1080...it does not :/

I do get it coming down from 1920x1080 to 640x360...but then, no matter how long I wait after a switch up to 1920x1080 again...it does not go back up. I get a ton of 'upstreamDiscarded' logged to the event logger though.

Can you offer any more advice? Thanks!

[christosts]

There may be a few parameters involved why the player doesn't step up. For example, the bitrate adaptation algorithm might decide to stay at low res because it thinks the available network throughput is not enough to switch from SD to HD.

Let's two do things:

1. Can you please share a patch with your recent changes? To make sure the code is set correctly.
2. Can you provide more context on your use-case? I understand you want to make the player select a specific resolution but what exactly are you trying to solve? There are a couple of different options how to change the player (each with its pros and cons), and at this point it's better to fully understand your use-case. For example, you could make canSelectFormat() return false for the lower resolution, or try to override the player's adaptation logic.

[google-oss-bot]

Hey @RenderHeadsSte. We need more information to resolve this issue but there hasn't been an update in 14 weekdays. I'm marking the issue as stale and if there are no new updates in the next 7 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

[google-oss-bot]

Since there haven't been any recent updates here, I am going to close this issue.

@RenderHeadsSte if you're still experiencing this problem and want to continue the discussion just leave a comment here and we are happy to re-open this.

[Ste-RH]

@christosts Apologies for the super slow response! Rushed out to Dubai for three weeks and then the holidays rampaged all over my time!

Here is the patch. I have left the code I was working on in LimitedAdaptiveTrackSelection:evaluateQueueSize, but commented out and it just returns zero. I assume this is correct for 'throw away all queued data please' ? I have removed the load controller buffering limits and it is all as default now.

Now the demo will change down to 640x360 at ~5 seconds, then at ~15 seconds it will remove the limit and 1920x1080 should be allowed again. No matter how long I leave the stream playing, it will never increase back up from 640x360 to 1920x1080.

Regarding use-case. I cannot offer one above 'the requirement is to seamlessly limit the stream to resolutions/bitrates'. This is because AVPro Video (Unity based video player asset) is used by a lot of developers for a vast variety of use-cases.

ExoPlayer_Issue9684_patch2.zip

[christosts]

Your override of evaluateQueueSize() returns 0 always. This has an implication to the bitrate adaptation logic which decides to not step-up because it thinks there is not enough buffer which increases risk of rebuffer. In your patch, I removed the override to evaluateQueueSize() and things started working. There is some delay in the algorithm response though:

• when limiting to SD resolution, the player will consume any HD chunks downloaded (and maybe it will download a few more as long as there is enough buffer) before it steps down to SD
• when allowing HD again, the player will start downloading HD but will also play any SD chunks that have been already downloaded (depends on the max buffer size)

Note that in your patch, you set the limit to SD 5 seconds. For the first 5 seconds the player will download whatever it thinks i's best. Note, you will not see immediate changes in the resolutions being played in general: the player has logic to smoothly transit from one resolution to another using a few criteria, including how many data are buffered.

If you want to start playing at low resolution immediately, set the limit before the player starts playing. If you want to step up, allow the higher-res formats to be selected and wait for the downloaded chunks to be played.

@tonihei can you offer some guidance how to implement evaluateQueueSize() if the app wants the player to step-up the resolution without consuming all buffered data? If it evaluateQueueSize() returns 0 always, AdaptiveTrackSelection will not step-up because the buffered duration is less than AdaptiveTrackSelection.DEFAULT_MIN_DURATION_FOR_QUALITY_INCREASE_MS?

[tonihei]

I don't think there is any other solution besides changing the parameters.

The main relevant parameter to determine whether switching up to a higher resolution is minDurationForQualityIncreaseMs. If there is less buffered data than this, the player will not switch up for the next loaded segment. If there is more buffer available than this threshold, the player will switch up (assuming enough network bandwidth is available). Note that this affects the next loaded segment only, so the player will always continue playing at least minDurationForQualityIncreaseMs of the old resolution because that's what is buffered already.

If you want to discard already buffered data AND switch up immediately as soon as possible, you need to decrease both minDurationToRetainAfterDiscard and minDurationForQualityIncreaseMs. Note that minDurationToRetainAfterDiscard should always be greater than minDurationForQualityIncreaseMs, otherwise the player discards data and can't switch up afterwards.

Aside, I think you don't necessarily need to override evaluateQueueSize because adjusting minDurationToRetainAfterDiscard is likely enough? This value can also be set dynamically, if you just override getMinDurationToRetainAfterDiscardUs()

[tonihei]

Closing as the question was probably answered.