search

w3c / mediacapture-record

MediaStream Recording
https://w3c.github.io/mediacapture-record/
Other
104 stars 21 forks source link

support dynamic format changes #173

Open guest271314 opened 5 years ago

guest271314 commented 5 years ago

The UA MUST record MediaRecorder/stream in such a way that the original video Track(s) width and height can be retrieved at playback time. The {{Blob}} containing entire recording returned when the recording finishes or smaller buffers of data returned at regular intervals containing the combination of all the original video Track(s) width and height MUST playback the input video Track(s) width and height.

The specification does not currently clearly state that input video track(s) width and height MUST be recorded and reflected in the Blob or smaller buffers of data recorded by MediaRecorder.

Currently at Chromium 76 neither variable video track input width and height are NOT output at the resulting Blob either by using canvas.captureStream(), HTMLMediaElement.captureStream() or WebRTC https://bugs.chromium.org/p/chromium/issues/detail?id=972470.

It is reasonable for users of MediaRecorder to expect variable width and height input to be reflected in the resulting media file(s).

This change fixes the omission in the specification as to what MUST occur as to recording and outputting a media file which contains variable witdth and and height input and makes it clear that the width and height of each input video track (image) MUST be recorded and output at both single Blob and multiple Blobs at dataavailable event.

Pull Request https://github.com/w3c/mediacapture-record/pull/172

alvestrand commented 4 years ago

Curious .... given that each encoded video frame already contains width and height .... what does it mean that "track width and height MUST be recorded"?

guest271314 commented 4 years ago

@alvestrand It means that if the input width and height are 150, 150, then then next frames are 300, 300, respectively, when the recording is played back the width and height are 150, 150, then 300, 300, not only the maximum width and height of all the input width and height, in this example case, 300, 300, with an inset in the video in the upper left-corner being 150, 150, with observable artifacts in the recorded video https://bugs.chromium.org/p/chromium/issues/detail?id=972470#c11 (https://bugs.chromium.org/p/chromium/issues/attachment?aid=404779&signed_aid=U4u0higdiNxaFBEPLgVbgg==&inline=1), https://bugs.chromium.org/p/chromium/issues/detail?id=972470#c12 (https://bugs.chromium.org/p/chromium/issues/attachment?aid=404780&signed_aid=PXpEFLruVo9ck6oul-PnCg==&inline=1). For the implementation to the degree MediaRecorder is related to or dependent upon <video> in the case of captureStream() from DOM element, both implementations should be fixed. Relevant to the specification it means as described above, for a comparative analysis, the result should be consistent with how Firefox implements video recording with MediaRecorder and video playback at <video> element, not exactly not how Chromium/Chrome currently implements those two components. For demonstration of the output described, see, in general https://bugs.chromium.org/p/chromium/issues/detail?id=972470. Does the above now have clear meaning to you?

Kindly contribute language that you can immediately understand, and that you believe others will immediately understand, if you have determined the language in this issue and linked PR https://github.com/w3c/mediacapture-record/pull/172 are not sufficient at described the issue and the language needed to fix the issue for specification requirements to be met.

guest271314 commented 4 years ago

@alvestrand Reproduction of the issue relevant to MediaStream implementation at Chromium browser, even without using MediaRecorder https://next.plnkr.co/edit/sMA653?p=preview&preview.

Note, the very basic code occasionally crashes the browser tab.

VP8 and VP9 implementation at Chromium do not appear to be configured to encode the correct pixel dimensions of input frames. See also https://bugs.chromium.org/p/chromium/issues/detail?id=983777.

guest271314 commented 4 years ago

@alvestrand The only way that have been able to output the identical pixel dimensions as input pixel dimensions at Chromium is to use h264 and avc1 (OpenH264) codecs at MediaRecorder MIME type https://plnkr.co/edit/Axkb8s?p=preview. That presents the issue of not being able to play the video at Mozilla Firefox or Nightly which does not support playback of .mkv files or if the extension is set to .webm will still detect the codecs and not play the media, further, h264 and avc1 codecs are not technically defined as being supported codecs for WebM.

guest271314 commented 4 years ago

@alvestrand Had concluded that Chromium implementation of HTML <video> element was the reason that variable frame width and height was not displayed at resulting WebM file output by MediaRecorder, given this code https://github.com/guest271314/MediaFragmentRecorder/blob/imagecapture-audiocontext-readablestream-writablestream/MediaFragmentRecorder.html displays only the initial width and height of the first frame or track width and height at Chromium, though the WebM file does display variable width and height at Mozilla Firefox and Nightly.

When the codecs are eithe VP8 or VP9 this code https://plnkr.co/edit/4JxS4O?p=preview demonstrates that the options set for VP8 and VP9 codecs at Chromium could be the issue, using https://github.com/thenickdude/webm-writer-js with the minimal amount of lines commented from the original repository to output variable width and height and dispatches resize event at HTML <video> element (though not sure yet how to set the correct frame rate using that repository).

MediaRecorder outputs a WebM video which for an unknown reason is constant width and height set at the initial frame or track.

There are use cases for both constant pixel dimensions and variable pixel dimensions. In this case the expected output is variable width and height (the same as input) which should cause HTML <video> element to dispatch resize event when the resolution changes.

That is what the linked PR https://github.com/w3c/mediacapture-record/pull/172 changes about this specification: the capability to set constant or variable pixel dimensions for the WebM file output, which has consistent display properties at both Mozilla Firefox and Nightly and Chromium and Chrome browsers, et al., instead of the inconsistent behaviour (between codecs and browsers implementing the same API) of the implementation of MediaRecorder relevant to pixel dimension encoding and output.