search

w3c / media-source

Media Source Extensions
https://w3c.github.io/media-source/
Other
268 stars 58 forks source link

Expose SourceBuffer as an ArrayBuffer #209

Open guest271314 opened 6 years ago

guest271314 commented 6 years ago

Currently there does not appear to be any means to get the media accumulated at a SourceBuffer at any time (https://github.com/w3c/html/issues/550, https://github.com/w3c/html/issues/555).

Provide some means of getting the media resource as a file (Blob having MIME type set the the codec set at MediaSource/SourceBuffer; ArrayBuffer) (ideally at any time, before .endOfStream() is called) at least when .endOfStream() is called.

jyavenard commented 6 years ago

What meaningful data do you expect to see in return? That's fine if you've only added the same content.

But if you've added different init segment, overwritten others, added multiple init segments for different resolution/encoding.

What then?

guest271314 commented 6 years ago

Not having viewed what a SourceBuffer following .endOfStream() looks like, am not sure what to expect.

What am relatively certain of now is that the accumulated data is not retrievable at .srcObject, nor any other way that have tried.

Would naively anticipate that a file having the MIME type equaling the codecs passed to MediaSource and .addSourceBuffer would be either be already assembled or capable of being assembled from the ArrayBuffers passed prior to calling .endOfStream().

The media can be seeked, and has a total duration at a <video> element following .endOfStream() - that accumulated data should be available as a discrete file object (ArrayBuffer; Blob).

The specification states

These URLs are the same as a Blob URI, except that anything in the definition of that feature that refers to File and Blob objects is hereby extended to also apply to MediaSource objects.

though the clear, unambiguous interpretation of that sentence, if accurate, would not have lead to this issue being filed - as a developer cannot request the Blob URL and get the data created by MediaSource object; instead 404 is response or 416 Range Not Satisfiable when trying to use the Blob URL in the same way as a Blob URL with Blob backing.

Is there a reason the SourceBuffer is not exposed?

guest271314 commented 6 years ago

The reason for this issue is that although have no experience using C++ browsed Chromium source code relevant to MediaSource and cannot locate where the data is stored.

Am trying to find a way to resolve https://github.com/w3c/media-source/issues/190 at Chromium, which still crashes the tab when trying to record a <video> element with HTMLMediaElement.captureStream(). Looking into the matter further is what led to https://github.com/w3c/html/issues/550 and https://github.com/w3c/html/issues/555.

The accumulated data is stored somewhere (https://beaufortfrancois.github.io/sandbox/media/source-buffer-limit.html). That data should be exposed to developers.

Am not sure if the original specification anticipated recording a <video> element where MediaSource is set at .src, though Firefox does not have an issue recording the media. If we have access to the accumulated data we might not need to use MediaRecorder at all - the data is already configured to be rendered at a <video> element.

If the decision is made to merge this issue with https://github.com/w3c/media-source/issues/190, that is fine.

guest271314 commented 6 years ago

Is 1.2 Definitions accurate as to the second paragraph below?

MediaSource object URL A MediaSource object URL is a unique Blob URI [FILE-API] created by createObjectURL(). It is used to attach a MediaSource object to an HTMLMediaElement.

These URLs are the same as a Blob URI [FILE-API], except that anything in the definition of that feature that refers to File and Blob objects is hereby extended to also apply to MediaSource objects.

guest271314 commented 5 years ago

@wolenetz When endOfStream() is executed can the SourceBuffer be exposed as an ArrayBuffer. Consider using changeType() to play variable resolution WebM videos. Currently due to a Chromium bug there is no way to record the <video> element using captureStream() without the tab crashing https://bugs.chromium.org/p/chromium/issues/detail?id=1001948 (likely the reason for the crash described at https://github.com/w3c/media-source/issues/190, et al.).

If SourceBuffer could be converted ideally to a WebM file available for download when the stream is complete that would be a viable workaround for using MediaRecorder.

What exactly is the data structure of a SourceBuffer when endOfStream() is called?

Is there any reason in the speification why a <video> with MediaSource set at src is restricted from being downloaded by the browser?

guest271314 commented 5 years ago

@wolenetz

Currently due to a Chromium bug there is no way to record the

Except using "video/webm;codecs=h264|avc1" or "video/x-matroska;codecs=h264|avc1" (https://plnkr.co/edit/Axkb8s?p=info) which are not capable of playback at Mozilla browsers.

wolenetz commented 2 years ago

This issue is part of a group of "introspection API" or similar API feature requests: #259, #209, #172, #40, #35.