Problem with adaptation sets that contain representations with different codecs

[barbibulle]

With an MPD that contains an audio adaptation set where there are two representations, each with a different codec, the player can play each of the representations by itself, but cannot switch from one to the other. I suspect that it doesn't reset the audio decoder, so feeding one elementary stream into a session setup with a different codec is likely to cause trouble. The DASH spec doesn't prohibit mixing codecs in the same adaptation set, and it would seem like the right thing to do if you want to offer a stream with a choice of AAC or EAC-3 (better than having two adaptation sets, one for each codec).

Here's an example: http://www.bok.net/tmp/dash/dual_codec/stream_aac.mpd

the same data with just the AAC codec: http://www.bok.net/tmp/dash/dual_codec/stream.mpd

and with just the EAC-3 codec (plays fine on Safari on El Capitan) http://www.bok.net/tmp/dash/dual_codec/stream_ec3.mpd

[LloydW93]

I believe I might not be entirely correct here, but 23009/1 does say of representations:

"If Representations are included in one Adaptation Set, and the client switches properly, the Media Presentation is expected to be perceived seamless across the switch. Clients may ignore Representations that rely on codecs or other rendering technologies they do not support or that are otherwise unsuitable."

In order to reset the audio decoder between those segments, the switch would then have to not be seamless. However, once one is chosen I think it may be correct behaviour for the client to see the other representation as "otherwise unsuitable" and prevent switching. Handling this case is potentially an improvement for the client.

[davemevans]

DASHIF IOP states that seamless switching is not expected across different formats. This is only mentioned in the multichannel section (9) but that is the only section where E-AC3 is allowed since it is not a base format (ie HEAAC)

DVB-DASH mandates codecs to be equivalent across Representations so that switching is seamless.

[barbibulle]

Seamless switching for audio tracks is often not going to be seamless, even without changing codecs. If you have an adaptation set with two different AAC representations (different bitrates for instance), you can't completely seamlessly switch from one to the other, since you'll have to reset your decoder (audio frames always depend on the previous frame because of the overlapped transforms). So it isn't any easier to switch from AAC to AAC than from AAC to EAC-3.

It would be a good idea for the next release of the DASHIF IOP to provide recommendation on interoperable ways to present multi-codec streams.

Using different adaptation sets for different codecs adds complexity, especially in being able to signal alignment between segments across adaptation sets.

In any case, I think that dash.js should handle these MPDs gracefully, either by refusing to switch once it picks a supported codec, or by selecting a "preferred" codec from the start.

[davemevans]

So it isn't any easier to switch from AAC to AAC than from AAC to EAC-3.

But it is because, for browser-based players, MSE handles the AAC to AAC switch for you. To switch formats you have to tear down and recreate the SourceBuffer and recreate since they are codec-specific, which will almost certainly cause an audible glitch.

[barbibulle]

Ok. Now, trying the same thing with two different adaptation sets, one for each codec, doesn't work any better. Using the reference player page on Safari/El Capitan, the stream starts with the E-AC3 adaptation set. If I manually switch to the AAC adaptation set using the drop-down menu, it stops with a MEDIA_ERR_DECODE. (MPDs with just one of the two codecs work fine by themselves)

Here's the MPD: http://www.bok.net/tmp/dash/dual_codec_2/stream.mpd

with AAC only (works) http://www.bok.net/tmp/dash/dual_codec_2/stream_aac.mpd

with E-AC3 only (works) http://www.bok.net/tmp/dash/dual_codec_2/stream_ec3.mpd

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because no further activity occurred. If you think this issue is still relevant please reopen it. Thank you for your contributions.