search

video-dev / hls.js

HLS.js is a JavaScript library that plays HLS in browsers with support for MSE.
https://hlsjs.video-dev.org/demo
Other
14.79k stars 2.57k forks source link

Chrome playback fails `The element has no supported sources.` with video with 2.1 channel aac audio #5481

Open campbellwmorgan opened 1 year ago

campbellwmorgan commented 1 year ago

What version of Hls.js are you using?

1.4.2

What browser (including version) are you using?

113.0.5672.93 (Official Build) (64-bit)

What OS (including version) are you using?

Windows 10

Test stream

No response

Configuration

const video = document.getElementById('vid');
        const hls = new HLS();
        hls.loadSource('./Output/master.m3u8');
        hls.attachMedia(video);
        hls.on(HLS.Events.MANIFEST_PARSED, () => {
            console.log('manifest parsed', video);
            document.getElementById('video').addEventListener('click', () => {
                video.play();
            })
        });

Additional player setup steps

No response

Checklist

Steps to reproduce

  1. Generate a fmp4 HLS stream from an MP4 with 2.1 channel AAC encoded audio a. Fragment the file using mp4fragment --fragment-duration 4000 (bento4) b. Generate HLS segments and manifests using mp4dash --verbose --hls --use-segment-timeline -o OutputDir <InputFile.mp4> (Bento4)
    1. Load main manifest
    2. Observe both audio and video segments being correctly loaded after manifest parsed
    3. Play the video

Expected behaviour

Video should play with audio in the browser. However, if audio is removed manually from main master.m3u8 manifest, the video plays correctly, but without audio.

See below example files for video that plays in browser and zip file with segmented hls files.

https://github.com/video-dev/hls.js/assets/1377367/4ccc35a2-5d43-41fa-9b03-f4276f37ae26

SegmentedHLS.zip

N.B worth noting that this HLS stream does play correctly in VLC if I open it via Network Stream

What actually happened?

Video does not play

Console output

`audio.mts:264 Uncaught (in promise) DOMException: The element has no supported sources.`

Chrome media internals output

RunSegmentParserLoop: stream parsing failed. append_window_start=0 append_window_end=inf
Error Group:
PipelineStatus
Error Code:
16
Stacktrace:
media\filters\chunk_demuxer.cc:1070
robwalch commented 1 year ago

The Multivariant Playlist says that the audio has 2 channels, but it is not stereo:

#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="audio",LANGUAGE="und",NAME="Unknown",AUTOSELECT=YES,DEFAULT=YES,CHANNELS="2",URI="audio/und/mp4a.40.2/media.m3u8"

HLS Authoring Guidelines say that you must include a stereo audio option. You cannot only inlcude 2.1 channel audio and claim that it is stereo.

robwalch commented 1 year ago

Not sure how you are getting a The element has no supported sources error. The browser fails to append the audio as provided, and as a result fails with "audio SourceBuffer error".

campbellwmorgan commented 1 year ago

@robwalch thanks for looking into this. I'm first assuming this is a bug with Bento4 which generated these files from my MP4 file. I have tried correcting the channels field in the m3u8 file to no avail (still get RunSegmentParserLoop: stream parsing failed. append_window_start=0 append_window_end=inf), but will try correcting the audio codec string too and get back to you

robwalch commented 1 year ago

From MediaCapabilities, this configuration should be supported:

navigator.mediaCapabilities.decodingInfo({
  type: 'media-source',
  audio: {
    contentType : 'audio/mp4; codecs="mp4a.40.2"',
    channels: 3, 
    bitrate : 421988,
    samplerate : 48000
  },
}).then((result) => {
  console.log(result);
}).catch((error) => {
  console.error(error);
});

// result
{powerEfficient: true, smooth: true, supported: true, keySystemAccess: null}

Unsure why Chrome produces an append error. I don't see anything in the media panel that is particularly helpful.

jprjr commented 1 year ago

One thing to note is using 2.1 channel audio in AAC requires using a channel index of 0 and defining the configuration in the AudioSpecificConfig (aka the global headers in init.mp4) via a program config element. Maybe there's an issue with that? That shouldn't be an issue (after all that's the whole point of having fragmented mp4, is to be able to provide those global headers) but it seems like I can't get Safari to play it, either, using native HLS.

I wonder if using a codec that can signal a 2.1 config without using global headers (I think AC3 can do this) would work.

Also worth mentioning:

The sample init.mp4 for the audio track lists a channel count of 2, not 3.

The program_config_element in the sample file configures the stream for a 3 channel layout of Front Left, Front Right, Back Center - not the 2.1 layout of Front Left, Front Right, LFE.

I thought maybe that meant something was wrong with the file (and there may be if you're expecting 2.1 audio) - but I made a 2.1 channel file with ffmpeg, with the expected LFE channel - and that fails to play with both hls.js and Safari.

I tried making a 3-channel file that can be represented in AAC without any program config elements (Front Center, Front Left, Front Right) - and that played fine across the board.

This may have something to do with AAC requiring program config elements for certain channel layouts?

jprjr commented 1 year ago

I suspect this may be a browser bug.

I made a series of multichannel test AAC files, and a web page that loads them using HTML audio, and again using Media source extensions.

On Firefox, all samples load and play regardless of method. But on Chrome, the samples that require a program config element fail when using Media Source Extensions.

I haven't tried Safari yet but I'm guessing it will behave similarly.

https://jprjr.github.io/multichannel-aac/

EDIT: I originally created buggy mp4 files (they used the wrong addressing mode), I've corrected that and update the link.

jprjr commented 1 year ago

Found an open bug on Chromium that matches this issue - https://bugs.chromium.org/p/chromium/issues/detail?id=241171 - so it's not specific to HLS.js. I believe Chromium is limited to the following channel layouts when using AAC in fragmented MP4, since these are the defined indexes:

Any other layout requires using Program Config Elements to signal.

One workaround may be using mpeg-ts. I haven't tried that yet.

robwalch commented 1 year ago

Found an open bug on Chromium that matches this issue - https://bugs.chromium.org/p/chromium/issues/detail?id=241171 - so it's not specific to HLS.js.

Yes. Thanks @jprjr! Same with:

robwalch commented 1 year ago

Adding the media-capabilities label even though adding MediaCapabilities checks does not solve the issue. Proper signaling and testing for support is needed on the Web platform to resolve these issues, otherwise it's a platform/media issue.

robwalch commented 1 year ago

Dropping from 1.5.0. #5704 will help by attempting to start on a stereo option and/or preferred codec if available as a default or auto-select option. The underlying issue however is that the media provided is not supported by the browser, and browser APIs available to test for support are not sufficient to signal that the media cannot be played.