HLS: Deduce bitrate order of multiple audio rendition groups based on usage in variants

[Sathish-Kumar-Kanagaraj]

I am using an hls url to play the video and also able to switch with multiple audios. I am getting surround and streo both the type of audio with same category language in the hls url. How can i change it to single language streaming and based on adaptive bit rate making it choose stereo or surround internally in the video. Kindly provide me with a solution.

Sample Link:https://bitdash-a.akamaihd.net/content/sintel/hls/playlist.m3u8

[ojw28]

The equivalent stereo and surround audios are already grouped into the same TrackGroups, which is what needs to happen to allow adaptation:

EventLogger:   Renderer:0 [
EventLogger:     Group:0, adaptive_supported=YES [
EventLogger:       [X] Track:0, id=0, mimeType=video/avc, bitrate=258157, codecs=avc1.4d400d, res=422x180, supported=YES
EventLogger:       [X] Track:1, id=1, mimeType=video/avc, bitrate=520929, codecs=avc1.4d4015, res=638x272, supported=YES
EventLogger:       [X] Track:2, id=2, mimeType=video/avc, bitrate=831270, codecs=avc1.4d4015, res=638x272, supported=YES
EventLogger:       [X] Track:3, id=3, mimeType=video/avc, bitrate=1144430, codecs=avc1.4d401f, res=958x408, supported=YES
EventLogger:       [ ] Track:4, id=4, mimeType=video/avc, bitrate=1558322, codecs=avc1.4d401f, res=1277x554, supported=NO_EXCEEDS_CAPABILITIES
EventLogger:       [ ] Track:5, id=5, mimeType=video/avc, bitrate=4149264, codecs=avc1.4d4028, res=1921x818, supported=NO_EXCEEDS_CAPABILITIES
EventLogger:       [ ] Track:6, id=6, mimeType=video/avc, bitrate=6214307, codecs=avc1.4d4028, res=1921x818, supported=NO_EXCEEDS_CAPABILITIES
EventLogger:       [ ] Track:7, id=7, mimeType=video/avc, bitrate=10285391, codecs=avc1.4d4033, res=4096x1744, supported=NO_EXCEEDS_CAPABILITIES
EventLogger:     ]
EventLogger:     Metadata [
EventLogger:       HlsTrackMetadataEntry
EventLogger:     ]
EventLogger:   ]
EventLogger:   Renderer:2 [
EventLogger:     Group:0, adaptive_supported=YES_NOT_SEAMLESS [
EventLogger:       [X] Track:0, id=stereo:English, mimeType=audio/mp4a-latm, codecs=mp4a.40.2, channels=2, sample_rate=48000, language=en, label=English, supported=YES
EventLogger:       [ ] Track:1, id=surround:English, mimeType=audio/mp4a-latm, codecs=mp4a.40.2, channels=2, sample_rate=48000, language=en, label=English, supported=YES
EventLogger:     ]
EventLogger:     Group:1, adaptive_supported=YES_NOT_SEAMLESS [
EventLogger:       [ ] Track:0, id=stereo:Dubbing, mimeType=audio/mp4a-latm, codecs=mp4a.40.2, channels=2, sample_rate=48000, language=dubbing, label=Dubbing, supported=YES
EventLogger:       [ ] Track:1, id=surround:Dubbing, mimeType=audio/mp4a-latm, codecs=mp4a.40.2, channels=2, sample_rate=48000, language=dubbing, label=Dubbing, supported=YES
EventLogger:     ]
EventLogger:     Metadata [

The problem is that ExoPlayer doesn't currently adapt audio and video at the same time (i.e. if video is going to adapt, then the audio track is fixed). Allowing adaptation of both simultaneously is tracked by #5902. @tonihei - You may or may not want to mark this as a duplicate.

Aside: In DASH and SmoothStreaming, it's generally up to the player to decide how to adapt audio and video together (i.e. the manifest allows the player to select any combination, and it's up to the player to decide what makes sense). HLS is different in that it specifies the allowed combinations in the master playlist. If you look at the master playlist linked above, it specifies that the surround audio track should be used if the video resolution is 958x408 or higher. I don't think this is a great design, since the player has contextual information that in practice affects which streams should be selected to provide the best user experience. For example the player typically knows what audio output is connected to the device. If the audio output is only stereo, then it would often be sensible for the player to use the stereo stream even for the highest video resolutions. The stereo stream will have been produced for stereo output by the content provider, and is likely to be better than downloading a surround sound stream and having the device down-mix it. So I think it's unlikely we'll adhere to the type of constraints that HLS master playlists are allowed to specify, even after #5902 is supported.

[tonihei]

The enhancement tracked by #5902 allows to automatically adapt both video and audio. However, our current experimental approach doesn't really work for this case for two reasons.

1. None of the audio tracks have bitrates associated with them. So it's unclear for a bitrate based algorithm how to select the tracks. We can use this issue to track at least basic support for this by just assuming that the order in which the audio groups are used in the variants corresponds to increasing bitrates. For example in this case, we can deduce that "surround" has a higher bitrate than "stereo" because it's used for the higher bitrate variants.

2. The surround tracks don't have the CHANNELS attribute set and ExoPlayer extracts the channels count from the actual sample stream. Because it happens to read the stereo stream, all formats (including the "surround" ones) are marked as having 2 channels only. If they were correctly annotated with 5 channels, then adaptation will not be possible because ExoPlayer doesn't consider two tracks with different numbers of channels as compatible for adaptation. We already have parameters to allow mixed-mime-type and mixed-sample-rate adaptation. So there is no particular reason not to allow mixed-channel-count adaptation as well. Please note that it may not make perfect sense though from an app' perspective as @ojw28 pointed out above.

[kpandroid]

@tonihei - you wrote:

"you can manually select any audio track for any video track." - could please point me where I can select right audio from new group after hls stream changes resolution?

[tonihei]

This statement was about making a manual selection of single tracks. See the DefaultTrackSelector documentation for more details on how to use SelectionOverrides.

[kpandroid]

@tonihei, do you mean that I can call SelectOverride and switch audio track from corresponded group without interruption of audio after a resolution adaptation? It seems I need something like GroupTrackSelection for audio tracks that connected to AdaptiveTrackSelection for video thats triggers GroupTrackSelection after updateSelectedTrack changes an index of videotrack.