Clarify expected content of "codecs" parameter in addSourceBuffer/isTypeSupported for streams with multiple tracks of same codec type

[wolenetz]

Let's say we have a multi-track webm stream containing 1 vp8 track and 2 vorbis tracks. Would a fully specified type parameter to addSourceBuffer()/isTypeSupported() for this stream be: 'video/webm; codecs="vp8, vorbis"'

or is the following required in a fully-specified type:

'video/webm; codecs="vp8, vorbis, vorbis"'?

Marking as vNext: For MSE v1, this issue is moot (see #137), since compliance with that version of the spec doesn't require fully specified codecs strings (though at least one implementation does, Chrome). Note that DASH manifests produced by at least one muxer implementation contain codecs strings containing the latter, undeduplicated codecs string, form (see https://codereview.chromium.org/2226443002/diff/360001/media/filters/media_source_state.cc#newcode597).

[wolenetz]

Furthermore, should the codecs in the codec string be in some specific order relative to the information in the initialization segment?

[jyavenard]

I don't see much the point to have to define every single codec found in the init segment, especially if it's present more than once.

But I also do think that having to always have the codec listed to be an unnecessary complicated burden. One we can do without. Afterall, all we need to know is the container type, not the codec

[dmlap]

From the perspective of a web author, I can say that requiring a fully-specified codecs string is a high threshold for many content producers. There's a ton of old content out there maintained by folks without a deep video background where details like the codecs haven't been recorded. Very few tools (even professional ones) expose codec specifications for content they support, or they require the user to piece together details from a bunch of different places.

I think the WHATWG wiki page on codecs is a good example of why this is tough for web authors. The section on codecs for MP4s says:

Video Codecs H.264 Baseline: avc1.42E0xx, where xx is the AVC level H.264 Main: avc1.4D40xx, where xx is the AVC level H.264 High: avc1.6400xx, where xx is the AVC level MPEG-4 Visual Simple Profile Level 0: mp4v.20.9 MPEG-4 Visual Advanced Simple Profile Level 0: mp4v.20.240

I don't think that knowing "AVC Level" means a hex-encoded string of a particular byte of the AVCConfigurationBox is part of the general web-developer knowledge set and there's a surprising lack of tool support for discovering this stuff.

[chcunningham]

A great tool to produce the full codec string is MP4Box.js. This is hosted here: http://download.tsi.telecom-paristech.fr/gpac/mp4box.js/filereader.html

To get the codec string, click the dropdown under "Load:" and choose "File". Then select your mp4. Then, under "File Overview" tab, you'll see the MIME, including the full codecs string.

[RobertWHurst]

@chcunningham Does this work with AVI, mpeg, webm or ogg?

I have absolutely no idea how to get the codec string consistently across file formats. Why do I need to specify these anyway, when the video element can detect this automatically?

I'm trying play chunks of a video files uploaded by our users. We're a digital signage company. I'm working with devices that have less than 2 gigs of memory, and I'm streaming 8K video. I'm using a service worker to cache the chunks. I wanted to simply use a video url and a service worker cache straight up without chunking, but partial content is not supported by service workers, so this is the approach I thought I could take, but needing the codec is a huge problem for me.

Does anyone have any advice they could share to help me move forward here?

[chcunningham]

@chcunningham Does this work with AVI, mpeg, webm or ogg?

AVI and Ogg are not formally supported by MSE. Mpeg has limited support (for mp3, no video). Because MSE allows for chunking of files, it was obligated to define what a valid chunk looks like and this was only done for handful of relatively modern (and conceptually compatible) file formats. Supported formats are described in the registry here.

I'm not aware of a mature js lib for parsing WebM, but there are a few leads when I google "matroska parser javascript".

Why do I need to specify these anyway, when the video element can detect this automatically?

Its mostly about failing sooner than later. MSE is meant to enable TV like experiences on the web. Sites that use it often have a number of encodings for each file and use the codecs string to ask the browser what will work without having to download and try (and fail) playback. Codec support varies between devices/browsers.

Does anyone have any advice they could share to help me move forward here?

If you're set on using MSE, you'll need to either require your clients to provide video in supported formats or transcode the video's after you receive them. For the latter, I'd look at ffmpeg.

[wolenetz]

Though the MIME type parameter to addSourceBuffer() (and now also the incubating SourceBuffer.changeType() API) can contain codec parameters to help obtain more proactive indication from the user agent about whether or not the codec parameters are supported, Chrome (through at least M70) currently requires codec parameters for those two APIs.

I expect to begin updating Chrome soon to relax its strict codec parameter requirement for addSourceBuffer() and changeType(), though apps should be aware that if they append media bytes for an unsupported codec, error will result later, perhaps as late as when playback reaches that portion of the buffered media timeline. Apps can obtain earlier indication of support by continuing to use full codec parameters in those two methods (and of course, in MediaSource.isTypeSupported()), and further, apps can use the Media Capabilities API (@chcunningham) to gain better information about the level of support.

Once its codec strictness for those two methods is relaxed, Chrome will be more compliant with MSE spec in this regard, and allow more flexibility to web apps. Note, the web app will still be required to indicate the bytestream type (e.g. "video/webm") at minimum, per MSE spec, for those two methods.

MediaSource.isTypeSupported() at least in Chrome (and IIUC, perhaps in other user agents - @jyavenard @jernoble) will continue to require strict codec parameters where necessary to disambiguate support, since that API yields only a true/false, without any "maybe" for the ambiguous cases, unlike htmlMediaElement.canPlayType().