Specify options or a method to encode pixel dimensions (width and height) of individual frames

[guest271314]

Background

MediaRecorder supports VP8 video codec at Chromium, Chrome, and Firefox. However, the specification provides no means to programmatically set encoder options, if available at the implementation source code, for width and height of individual frames (images) of input. The result at Chromium and Chrome is that "video/webm;codecs=vp8" and "video/webm;codecs=vp9" result in a WebM file that does not match the input width and height where the input frames potentially have variable width and height (https://bugs.chromium.org/p/chromium/issues/detail?id=972470; https://bugs.chromium.org/p/chromium/issues/detail?id=983777). The only code shipped with Chromium that have been able to record variable width and height input which outputs frames identical to the input frames is using "video/x-matroska;codecs=h264" or "video/x-matroska;codecs=avc1" https://plnkr.co/edit/Axkb8s?p=info although technically WebM is specified to only support VP8 or VP9, the fact is Chromium, Chrome support video codecs other than VP8 or VP9 for WebM container (https://bugzilla.mozilla.org/show_bug.cgi?id=1562862; https://bugs.chromium.org/p/chromium/issues/detail?id=980822; https://bugs.chromium.org/p/chromium/issues/detail?id=997687; https://bugs.chromium.org/p/webm/issues/detail?id=1642). When the codecs are changed to VP8 or VP9 the resulting WebM file does not output the correct pixel dimensions corresponding to input MediaStreamTrack.

Mozilla Firefox and Nightly does record and encode the correct variable input video frames, both when using MediaRecorder to create media files and MediaSource to playback media files.

Proposed solution

Specify options or a method to encode pixel dimensions (width and height) of individual frames, and make sure the options and/or method does output the expected result, if not then write code from scratch that achieves that requirement to be included in Web Codecs specification.

For example, using code at the Explainer

const videoEncoder = new VideoEncoder({
  codec: "vp9", 
  // code
});

include options or a method to explicitly set the encoder to encode each input frame width and height, to avoid the output at Chromium, Chrome, which outputs pixel dimensions that do not match input dimensions.

[guest271314]

After various tests had concluded that the issue with Chromium browser not displaying variable pixel dimension encoded WebM files output by MediaRecorder was HTML <video> element implementation, as this code https://github.com/guest271314/MediaFragmentRecorder/blob/imagecapture-audiocontext-readablestream-writablestream/MediaFragmentRecorder.html outputs pixel dimensions of the initial width and height, and is not resized at Chromium, though is resized at Mozilla Firefox.

However, this code https://plnkr.co/edit/4JxS4O?p=preview which uses a slightly modified version of https://github.com/thenickdude/webm-writer-js although the frame rate is not correct displays variable width and height and dispatches resize event.

Developers should have the ability to pass options to the encoder which either encode a single initial width and height and does not resize, or output variable width and height and resizes.

[pthatcherg]

WebCodecs (as currently designed) does not do containerization. It does encode/decode and the JS/wasm can choose how to containerize. Thus, you don't need to work around limitations/bugs in various browsers related to container formats because the JS/wasm would be responsible for doing so.

For example, you could use WebCodecs to encode VP8 and then use your own JS to create the WebM container with whatever settings you want in the container. WebCodecs settings would only be for things related to the encoding, not the containerization.

[guest271314]

@pthatcherg This issue does not address "containerization". This issue is intended to address settings passed to the codec encoder.

[pthatcherg]

Sorry if I misunderstood. Could you perhaps give an example of what you think the parameters would look like? We have definitely have a mechanism for setting parameters (see here).

It sounds like you want one for setting the dimensions. But what if the dimensions of the input frame don't match the parameters? Do you want the codec to scale before encoding?

[guest271314]

Re

WebCodecs (as currently designed) does not do containerization

What is "muxer" intended to mean at the explainer, other than combining audio, video, subtitles, etc. into a single output?

But what if the dimensions of the input frame don't match the parameters?

Not sure what you mean. The concept is to control the entire process.

[pthatcherg]

The "muxer" in the example is meant to be JS code, not an API from the browser. In fact, "Direct APIs for media containers (muxers/demuxers)" is listed as a non-goal.

If you create a video encoder configured to 720p resolution and feed it a frame that's 1080p, would do you expect it to do?

[guest271314]

It depends on the requirement. One use case might be to scale all frames to 720, which Chromium <video> does at output. Another use case might be to read and encode each frame to the input image pixel dimensions. Expect this API to do exactly what the initial and/or dynamic configuration settings state.

[pthatcherg]

It sounds like what you're asking for is a "scaleBy" parameter which defaults to 1 (no scaling). There's a similar parameter in WebRTC that allows scaling before encoding.

[guest271314]

Am asking that all parameters that the encoder used be capable of being set both initially and dynamically. Note also that contraints that are specified for getUserMedia() do not apply to MediaStreamTracks, for example from canvas.captureStream() or HTMLMediaElement.captureStream(). Though the fact might not be immediately clear, each individual specification which uses MediaStreamTracks (e.g., ImageCapture, canvas.captureStream(), getDisplayMedia(), HTMLMediaElement.captureStream()) need to define their own constraints, both what is expected and what is not expected. This is clear at Firefox, where HTMLMediaElement.captureStream().getVideoTracks()[0].getSettings() returns an empty JavaScript plain object {}, while getSettings() on a MediaStreamTrack from getUserMedia() returns the constraints of the track, e.g., see https://bugzilla.mozilla.org/show_bug.cgi?id=1537986.

[pthatcherg]

Oh.....

Yes, setting parameters dynamically will be possible. Doing that properly is what https://github.com/pthatcherg/web-codecs/issues/5 is about.

[guest271314]

Yes, that could resolve part of the issue. Since WritableStream gets data from ReadableStream, both parts of the transform stream need to have the capability to dynamically change the setting during the read/write.

[pthatcherg]

Which part of the issue is not resolved by dynamically changing the settings?

[guest271314]

@pthatcherg

WebCodecs (as currently designed) does not do containerization. It does encode/decode

Does WebCodecs intend to write and use its own video decoder? Or rely on an existing decoder at the implementation?

[chcunningham]

The spec offers a mechanism to configure (and reconfigure) encoder sizes. https://wicg.github.io/web-codecs/#dictdef-videoencoderconfig

The intent is that an implementing UA must actually produce frames at the configured size.

Feel free to re-open if more to discuss.