Can't play Kaltura multiplexed HLS content that looks like non-multiplexed content

[bansalkirti]

Have you read the FAQ and checked for duplicate open issues? yes

What version of Shaka Player are you using? 3.0.8

Can you reproduce the issue with our latest release version? Yes

Can you reproduce the issue with the latest code from master? Yes

Are you using the demo app or your own custom app? The issue is reproducible with both

If custom app, can you reproduce the issue using our demo app? The issue is reproducible with both

What browser and OS are you using?

For embedded devices (smart TVs, etc.), what model and firmware version are you using? N/A

What are the manifest and license server URIs? sent you the URI on shaka-player-issues@google.com Email Subject: GitHub Issue 3149

What did you do? Play the stream using shaka player.

What did you expect to happen? The stream should have played

What actually happened? failed to get buffered range for audio DOMException: Failed to read the 'buffered' property from 'SourceBuffer': This SourceBuffer has been removed from the parent media source.

Logfile shaka-player-demo.appspot.com-1612887429970.log

Console Screen Shot

[ismena]

HI @bansalkirti thanks for the report, we received the manifest and will look at it this week.

[ismena]

Alright! So :) Here's what we think is happening: your manifest has a set of audio MEDIA tags and a set of variant tags (let me know if you're not familiar with the HLS manifest structure, I can expand on that sentence):

#EXTM3U
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aac",NAME="undefined a1",DEFAULT=YES,AUTOSELECT=YES,LANGUAGE="undefined"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aac",NAME="undefined a2",DEFAULT=NO,AUTOSELECT=YES,LANGUAGE="undefined",URI="tracks-a2/mono.m3u8"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aac",NAME="undefined a3",DEFAULT=NO,AUTOSELECT=YES,LANGUAGE="undefined",URI="tracks-a3/mono.m3u8"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aac",NAME="undefined a4",DEFAULT=NO,AUTOSELECT=YES,LANGUAGE="undefined",URI="tracks-a4/mono.m3u8"
#EXT-X-STREAM-INF:AUDIO="aac",CLOSED-CAPTIONS=NONE,RESOLUTION=640x360,FRAME-RATE=25.000,CODECS="avc1.64001e,mp4a.40.2",AVERAGE-BANDWIDTH=1750000,BANDWIDTH=2190000
tracks-v4a1/mono.m3u8
#EXT-X-STREAM-INF:AUDIO="aac",CLOSED-CAPTIONS=NONE,RESOLUTION=1024x576,FRAME-RATE=25.000,CODECS="avc1.64001f,mp4a.40.2",AVERAGE-BANDWIDTH=2980000,BANDWIDTH=3720000
tracks-v3a1/mono.m3u8
#EXT-X-STREAM-INF:AUDIO="aac",CLOSED-CAPTIONS=NONE,RESOLUTION=1280x720,FRAME-RATE=25.000,CODECS="avc1.64001f,mp4a.40.2",AVERAGE-BANDWIDTH=4510000,BANDWIDTH=5640000
tracks-v2a1/mono.m3u8
#EXT-X-STREAM-INF:AUDIO="aac",CLOSED-CAPTIONS=NONE,RESOLUTION=1920x1080,FRAME-RATE=25.000,CODECS="avc1.640028,mp4a.40.2",AVERAGE-BANDWIDTH=6040000,BANDWIDTH=7550000
tracks-v1a1/mono.m3u8`

The audio tags are audio-only content, while the variant tags are actually multiplexed audio+video. Because the variant tracks are marked with AUDIO="aac", we pair the tracks together, assuming that the audio tags are audio and the variant tags are video. In reality, though, variant tags are multiplexed content and when we feed them to the browser as video, it errors out.

What you can do is either

• get rid of the association between the MEDIA tags and the VARIANT tags (delete the AUDIO="aac" thing)
• just drop the audio only group altogether. We don't support switching between audio only and audio+video on browsers other than Safari anyway

Let me know if you have any questions!

[bansalkirti]

@ismena many thanks for explaining. Let me take it to my team. Will trouble you again if any question.

thanks again for such a nice reply

[bansalkirti]

@ismena However, it's playing on Firefox. Are you handling firefox differently? (recording attached)

https://user-images.githubusercontent.com/65708547/107652636-cc20ba80-6ca6-11eb-8b08-6fd6679f0a6f.mov

[ismena]

We don't! My guess is, Firefox just handles this scenario better and is not as picky about label vs actual content. Let me tag @joeyparrish (who is way smarter than yours truly, but also much busier, so he may or may not reply :p) and see if he has a better guess.

[bansalkirti]

@ismena I am quite confused now as what I knew was that parsing and fetching data is handled by shaka and it just passes the buffer to the browser API to play.

firefox/chrome does not support HLS natively, it's shaka that is doing everything from fetching to parsing to adaptive playback.

so how it's playing on firefox and not on chrome...

🤔

thanks again for helping

[joeyparrish]

@ismena I am quite confused now as what I knew was that parsing and fetching data is handled by shaka and it just passes the buffer to the browser API to play.

firefox/chrome does not support HLS natively, it's shaka that is doing everything from fetching to parsing to adaptive playback.

so how it's playing on firefox and not on chrome...

You are correct. We parse the metadata from the playlist, fetch the segments, and feed them to the browser.

What the player understands from the playlist is "these segments are video only" and "these are audio only", and so when we set up the MediaSource and SourceBuffer objects (browser-provided API), we tell the browser to expect that, too.

So everything we do in the player is the same on both browsers.

However, it appears that Chrome rejects the actual content segments if they don't match the configuration, but Firefox does not. This is a matter of the browser's MediaSource implementation, and out of our control. Firefox seems to tolerate this mismatch, but Chrome does not.

Does this help explain?

[bansalkirti]

Thanks, @joeyparrish for sparing time to reply from your busy schedule.

I understand that is something that depends on the browser. However, this is also true that HLS.js, JW Player, and other players are able to play the stream on chrome. So, I am in dilemma, we are getting such third-party streams and cannot alter them.

Is it possible for you to raise it as an enhancement change request? I know that I am asking more :) however just in case if it possible.

[mardu]

I am having the exact same issue with Kaltura nginx-vod-module HLS playlists. As the setting vod_hls_force_unmuxed_segments is by default off all the video variants and segments are multiplexed. Everything works perfectly with only one audio track. Then the multiplexed variant is used and no separate audio variants are created.

A default variant .m3u8 files looks like this /index-v1-a1.m3u8. After changing it manually to /index-v1.m3u8 the video starts playing in Chrome.

I was wondering if it's somehow possible to preprocess the master.m3u8 playlist before the player loads it?

[joeyparrish]

@joeyparrish wrote:

What the player understands from the playlist is "these segments are video only" and "these are audio only", and so when we set up the MediaSource and SourceBuffer objects (browser-provided API), we tell the browser to expect that, too.

However, it appears that Chrome rejects the actual content segments if they don't match the configuration, but Firefox does not. This is a matter of the browser's MediaSource implementation, and out of our control. Firefox seems to tolerate this mismatch, but Chrome does not.

@bansalkirti wrote:

I understand that is something that depends on the browser. However, this is also true that HLS.js, JW Player, and other players are able to play the stream on chrome.

I am not an expert in those projects, but I can guess that they are doing one of two things:

1. Demuxing/remuxing the content, so that the browser consumes video and audio separately
2. Correctly interpreting the multiplexed content as such, and telling the browser what to expect in advance

Option 1 would require architectural changes in Shaka Player. We just aren't set up to do that, and it's not in our plans at this time.

Option 2 would be ideal. We could ignore the audio-only streams, because we don't support a mix of audio-only and audio+video tracks. If we knew that the audio+video streams were, in fact, files that contained both, we already have logic to do that.

The problem is that the playlist itself has literally no clues to tell us that this is the case. As far as I can tell, it looks identical to content in which audio and video streams are strictly separate.

It says CODECS="avc1.640028,mp4a.40.2", which could in general mean either "this contains both avc1.640028 video and mp4a.40.2 audio" or "this is avc1.640028 video, with an associated audio group containing mp4a.40.2 audio". The existence of the AUDIO="aac" attribute is how we differentiate, and it leads us to the wrong conclusion. But there is honestly no other reasonable conclusion we could reach from the playlist alone.

Now, other players that deal with the container can discover at demux time that the stream contains both audio + video, and it can react then. But we don't do that. The browser does that for us.

Native players generally deal with content at the container level. And web-based players can, but we choose not to duplicate in JavaScript work that the browser is meant to do in this environment. In fact, web-based players that choose to demux content for any reason must then remux it before feeding it to the browser. This is a waste of the client's resources, in our opinion, even though it does prevent us from being capable of playing content such as this.

@mardu wrote:

I am having the exact same issue with Kaltura nginx-vod-module HLS playlists. As the setting vod_hls_force_unmuxed_segments is by default off all the video variants and segments are multiplexed. Everything works perfectly with only one audio track. Then the multiplexed variant is used and no separate audio variants are created.

It sounds to me as if the default should be changed. Maybe that's something worth bringing up with Kaltura, @mardu, or with whomever is sending you those streams, @bansalkirti.

@mardu wrote:

I was wondering if it's somehow possible to preprocess the master.m3u8 playlist before the player loads it?

Absolutely! You can use something called a "response filter" in Shaka Player to manipulate the content of the playlist before we parse it. There is an example of a response filter in one of our tutorials, though it's for a completely different purpose. You can check out the "license wrapping" tutorial for details.

In essence, you can register a response filter, check for the request type shaka.net.NetworkingEngine.RequestType.MANIFEST, then extract the text of the playlist from response.data. You can then modify the playlist text, which can then be encoded back to Uint8Array and assigned back to response.data.

I hope this helps!

[joeyparrish]

Closing due to inactivity. If you need to reopen this issue, just put @shaka-bot reopen in a comment. Thanks!