mimeType ambiguity: "video/webm;codecs=vp8" means?

[jan-ivar]

(Web compat issue from https://github.com/web-platform-tests/wpt/pull/20530). The following works in Chrome:

const stream = await navigator.mediaDevices.getUserMedia({video: true, audio: true});
const rec = new MediaRecorder(stream, {mimeType: 'video/webm;codecs=vp8'});
rec.start();

...but fails in Firefox with NotSupportedError: MediaRecorder.start: An audio track cannot be recorded: video/webm;codecs=vp8 indicates an unsupported codec

If we add opus then it works in both:

const stream = await navigator.mediaDevices.getUserMedia({video: true, audio: true});
const rec = new MediaRecorder(stream, {mimeType: 'video/webm;codecs=vp8,opus'}); //<--
rec.start();

Who's right? The spec isn't super clear. They should work the same. The type is valid:

MediaRecorder.isTypeSupported('video/webm;codecs=vp8') // true

...so video/webm;codecs=vp8 is a valid mimeType passed to MediaRecorder. But does it mean:

1. Don't record audio, or
2. Use default codec for any audio?

https://tools.ietf.org/html/rfc6381#section-3.1 says "When the 'codecs' parameter is used, it MUST contain all codecs indicated by the content present in the body part", but also calls it "informative".

Chrome fails to include opus in the resulting blob type, so that's a bug for sure I think.

But mimeType is historically a descriptor for a body, while in MediaRecorder, we're using it instead as a selector, which may be uncharted territory.

E.g. 'video/' in a mimeType descriptor may imply audio. Does it also imply audio as a selector?

[jan-ivar]

I'm updating the tests to avoid the ambigous mimeTypes for now, but it remains a web compat issue.

[guest271314]

But does it mean:

1. Don't record audio, or

Yes. When affirmative effort is put forth in code to set a mimeType at MediaRecorder that setting must be controlling, not up to the application to decide what to do. Otherwise mimeType could be omitted from the code altogether, which requires less effort than taking the time to set specific properties at the options object.

Firefox can set the MediaStreamTrack of kind "audio" internally to muted or enabled to false for the case of "video/webm;codecs=vp8"

If any Track within the MediaStream is muted or not enabled at any time, the UA will only record black frames or silence since that is the content produced by the Track.

Chrome fails to include opus in the resulting blob type, so that's a bug for sure I think.

Chromium 81 includes "opus" in the Blob type.

Blobs can have no type set yet still reference the same underlying data. Would not rely on a Blob or File type for accurate reflection of the contents of the file at the time of the getting Blob.type.

Should Chrome record audio when the author of the code explicitly set mimeType omitting an audio codec? From perspective here, no. Should Firefox throw an error when the MediaStream set at MediaRecorder contains video codec and both video and audio MediaStreamTracks? No. The specification section above provides a means to record the explicitly set codecs, by treating the track as mute or not enabled.

One case for not treating mimeType as explicit and nothing more or less is if the codec were not VP8 (default), but VP9, where the user might have the requirement to use VP9 instead of VP8, for example, to reduce file size by the difference between the two codecs, while still expecting audio to be default Opus.

One approach to resolve the potential for ambiguity and opposite yet equally reasonable expectations is to implement constraints similar to {mimeType:video/webm;codecs=vp8, codecs:"exact /* audio track not recorded */|default/* audio track recorded */"}. If the tracks are not present, e.g., {audio:true, video:false}, {mimeType:"video/webm;codecs=vp8"} set to default without an error. Basically, provide some very simple means for the code author to indicate if settings are explicit and not anything else or should revert to default if one or more track and codecs setting and MediaStreamTracks in the MediaStream are not uniform.

[guest271314]

One existing model that can be considered for comparison is addSourceBuffer() of Media Source Extensions. If "video/webm;codecs=vp8" is set without ",opus" the audio will not play the media at Chromium. Firefox will play the audio. Another compat issue. Chromium behaviour expects the parsers to be used to be explicitly set when passing codec to addSourceBuffer(). There does not appear to be a bright line between what users are expecting from the implementation by default and the requirement to explicitly set codecs. FWIW prefer the latter: the user explicitly sets values for the code based on well defined specification that clearly states default and non-default behaviour.

[Pehrsons]

The spec says:

8. If the [[ConstrainedMimeType]] slot specifies a media type, container, or codec, then run the following sub steps:

   8.1. Constrain the configuration of recorder to the media type, container, and codec specified in the [[ConstrainedMimeType]] slot.

   8.2. For each track in tracks, if the User Agent cannot record the track using the current configuration, then throw a NotSupportedError DOMException and abort all steps.

My intention when writing this was that a present codecs parameter locks the codecs in. As in, the vp8 codec is specified, thus the UA cannot add another. In the end, the vp8 codec alone can not record an audio track, thus we should throw.

In my mind the main reason in favor of this behavior is that web developers may catch bugs early. Because they're specifying codecs they want exactly those codecs -- trying to record mismatched tracks is likely a bug on their part.

On the other hand, if the UA is allowed to add codecs, would it be allowed to add a codec of the same type as the one already present? I'd argue no, because it is already specified. Applying the same logic to the entire codecs parameter reduces the corner cases and simplifies the logic, which seems like an improvement.

Can we 'just' clarify the spec here?

[jan-ivar]

A converse case is bug 1881826 where specifying { mimeType: "video/webm;codecs=vp8,opus" } will still record video-only MediaStreams in Chrome, but not Firefox.

Compounding the foot-gun: whether a mediaStream from navigator.mediaDevices.getDisplayMedia() contains an audio track or not may depend varying factors, even user choice. Reporters found it "counterintuitive to have to check the stream for an AudioTrack in order to provide the right MimeType string."

Might it be less surprising if a MediaRecorder just ignores a codec it doesn't end up needing?

[jan-ivar]

If so, then back to the OP problem:

Might it be less surprising if MediaRecorder's mimeType setting (to basically override UA codec choices) didn't also act as an input selector?

A way to exclude audio from a recording already exists:

  const rec = new MediaRecorder(new MediaStream(...stream.getVideoTracks()), options);

...and works the same whether options are passed in or not.

[dontcallmedom-bot]

This issue had an associated resolution in WebRTC March 26 2024 meeting – 26 March 2024 (mimeType ambiguity: "video/webm;codecs=vp8" means?):

RESOLUTION: Align spec with Chrome