Open guest271314 opened 6 years ago
Just tried with https://github.com/w3c/web-platform-tests/blob/master/media-source/mp4/test.mp4 and the .mp4
mentioned at OP, which properly plays the two files in sequence.
However, if we place the array buffer of media having shortest duration before array buffer of media having longest duration the media playback stops rendering after first media file playback completes, though .currentTime
of media is three seconds longer than shortest media file.
Followup to previous post, if longest media is placed after shortest media within SourceBuffer
and after shortest media stops playback we can seek to longest part of media, which plays the remainder of expected media.
Have you experimented with different MSE implementations? Is there any chance this is an implementation bug in the browser you are using?
/paulc HME WG Chair
@paulbrucecotton Trying at Chromium 60.0.3112.78 (Developer Build). Have not tried at Firefox, yet. Seeking guidance on what the recommended or working pattern is to achieve expected result.
Can include the code tried here if that will be helpful.
@paulbrucecotton Just tried at Firefox 55, though the codec video/mp4; codecs=avc1.42E01E, mp4a.40.2
does not appear to be supported.
@paulbrucecotton Code tried using fetch()
instead of <input type="file" multiple>
http://plnkr.co/edit/KBbopiad1wR25nqtrvxw?p=preview
<!DOCTYPE html>
<html>
<head>
</head>
<body>
<br>
<video controls="true" autoplay="true"></video>
<script>
(async() => {
const mediaSource = new MediaSource();
const video = document.querySelector("video");
// video.oncanplay = e => video.play();
const urls = ["https://nickdesaulniers.github.io/netfix/demo/frag_bunny.mp4", "https://raw.githubusercontent.com/w3c/web-platform-tests/master/media-source/mp4/test.mp4"];
const request = url => fetch(url).then(response => response.arrayBuffer());
// `urls.reverse()` stops at `.currentTime` : `9`
const files = await Promise.all(urls.map(request));
/*
`.webm` files
Uncaught DOMException: Failed to execute 'appendBuffer' on 'SourceBuffer': This SourceBuffer has been removed from the parent media source.
Uncaught DOMException: Failed to set the 'timestampOffset' property on 'SourceBuffer': This SourceBuffer has been removed from the parent media source.
*/
// const mimeCodec = "video/webm; codecs=opus";
// https://stackoverflow.com/questions/14108536/how-do-i-append-two-video-files-data-to-a-source-buffer-using-media-source-api/
const mimeCodec = "video/mp4; codecs=avc1.42E01E, mp4a.40.2";
const media = await Promise.all(files.map(file => {
return new Promise(resolve => {
let media = document.createElement("video");
let blobURL = URL.createObjectURL(new Blob([file]));
media.onloadedmetadata = async e => {
resolve({
mediaDuration: media.duration,
mediaBuffer: file
})
}
media.src = blobURL;
})
}));
console.log(media);
mediaSource.addEventListener("sourceopen", sourceOpen);
video.src = URL.createObjectURL(mediaSource);
async function sourceOpen(event) {
if (MediaSource.isTypeSupported(mimeCodec)) {
const sourceBuffer = mediaSource.addSourceBuffer(mimeCodec);
for (let chunk of media) {
await new Promise(resolve => {
sourceBuffer.appendBuffer(chunk.mediaBuffer);
sourceBuffer.onupdateend = e => {
sourceBuffer.onupdateend = null;
sourceBuffer.timestampOffset += chunk.mediaDuration;
console.log(mediaSource.duration);
resolve()
}
})
}
mediaSource.endOfStream();
}
else {
console.warn(mimeCodec + " not supported");
}
};
})()
</script>
</body>
</html>
One issue that have been facing is testing sourcing and using properly encoded media files served with CORS headers. Not sure if files created using MediaRecorder
are encoded accurately.
@guest271314 wrote:
@paulbrucecotton Just tried at Firefox 55, though the codec video/mp4; codecs=avc1.42E01E, mp4a.40.2 does not appear to be supported.
What do you mean it's not supported? it certainly is https://jsfiddle.net/hcfvyx9k/
[removed off topic comments]
@jyavenard
What do you mean it's not supported? it certainly is https://jsfiddle.net/hcfvyx9k/
The quotes surrounding value of codec
key are the difference. Including approproiate quotes
'video/mp4; codecs="avc1.42E01E,mp4a.40.2"'
returns expected result at Firefox 55.
Still have yet to try pattern with more than two media files.
@jyavenard Just tried with four requests for media files. Chromium 60 plays the files in sequence without an issue using same code pattern. Firefox 55 logged an apparent fetch()
error
TypeError: NetworkError when attempting to fetch resource. (unknown)
http://plnkr.co/edit/9FYe4cJ6d4BC0B0LyOmN?p=preview, https://jsfiddle.net/hcfvyx9k/1/
@jyavenard The media finally rendered playback at Firefox 55 with four requests. Not sure about the reason for previous errors? Timeout?
@guest271314 you had a space in your original mimetype separating the two codecs, so yes, quotes are required then. If you remove the space, there's no need for quotes.
Your fiddle worked first go here. Glad it's working, but that's not using sequence mode as far as I can tell. As far as network errors are concerned, looking into the network devtools (Command+Option+E on mac) would give you more details
@jyavenard
Glad it's working, but that's not using sequence mode as far as I can tell.
Not sure what you mean as to sequence mode? What is the effective rendering difference between sequence and segments mode relevant to the expected result described at OP? Can you illuminate?
Oops my bad. I read the title and understood it as you using sequence mode. My first comment was about using the source buffer in sequence mode (as opposed to segment mode)
@jyavenard
My first comment was about using the source buffer in sequence mode (as opposed to segment mode)
What is the effective difference between the two modes as to processing or rendering the media? Is "segments"
the default mode?
http://w3c.github.io/media-source/#dom-mediasource-addsourcebuffer step 7 for the default value.
For an explanation of the different modes
@jyavenard Utilizing https://github.com/legokichi/ts-ebml was able to convert a Blob
retrieved at dataavialable
event of MediaRecorder
recording the playback of a media fragment to a Blob
where when passed to URL.createObjectURL()
and set at an HTMLMediaElement
has .duration
set. Had the sense that that would then allow MediaSource
to render the ArrayBuffer
representation of Blob
having metadata written. However, errors are thrown emitted from SourceBuffer
Uncaught DOMException: Failed to set the 'timestampOffset' property on 'SourceBuffer': This SourceBuffer has been removed from the parent media source. at SourceBuffer.sourceBuffer.onupdateend.e
.
The use case is trying to play media fragments at MediaSource
, which appears to have the capability to render media playback with minimal apparent gap between discrete buffers.
At step 6. at first link does the algorithm expect the metadata within the file to be at a certain location within the file being read?
RE: https://bugs.chromium.org/p/chromium/issues/detail?id=820489
I'm also a bit confused on the use case: is it meant to accommodate varying mime-type/codec media fragements' concatenation into a single resulting stream for later use? Or is there some other purpose.
MediaRecorder
was used to (presumably) create a single type of file structure (EBML, webm) which could be parsed by MediaSource
.
MediaSource
was used because we cannot currently render multiple tracks at a <video>
or <audio>
element "seamlessly". The closest have been able to come is with audio, either concatenating Blob
s to a single webm
file or using Web Audio API OfflineAudioContext
(which have not yet done so consistently).
Where we want to play specific segments of media content as a single stream to convey the resulting communication when such media is "concatenated" or "merged", and not rendered individually, or with no noticeable (to perception) gaps in in the media playback.
The concept is to, ideally
MediaSource
or MediaStream
or rendering at <canvas>
, etc.)
c. create a single file of the media fragments as a single media file (for example, for download);
which incorporates both immediate usage of the media and later usage of the media;<video>
element at the browser can play the file type, we should be able to re-arrange the content in any conceivable manner at the browser - without using third-party polyfils or libraries;document
; for example at a Worker
thread; as we want to splice, re-arrange, concatenate, merge the media content as raw data, then either offer the file for download, stream the media as the normalization is occurring, or post the resulting media to a document
for rendering.The above is a brief synopsis of the concept. Based on the combined implementations of OfflineAudioContext
, MediaSource
and MediaRecorder
; a conceptual OfflineMediaContext.
To a an approeciable extent the use case is possible using <canvas>
and AudioContext
and OfflineAudioContext
, (see https://stackoverflow.com/q/40570114 ; https://bugs.chromium.org/p/chromium/issues/attachmentText?aid=328544) though the video media presently needs to be rendered first.
Am not sure if there is interest in a unified specification to merge the disparate APIs into a single "media editing" API, or if there is any interest at all in the use case outside of creating a "playlist" of media.
The code could look something like
<!DOCTYPE html>
<html>
<head>
</head>
<body>
<script>
(async() => {
let audioContext = new AudioContext();
let urls = [{
src: "https://upload.wikimedia.org/wikipedia/commons/a/a4/Xacti-AC8EX-Sample_video-001.ogv",
from: 0,
to: 4
}, {
src: "https://mirrors.creativecommons.org/movingimages/webm/ScienceCommonsJesseDylan_240p.webm",
from: 10,
to: 14
}, {
from: 55,
to: 60,
src: "https://nickdesaulniers.github.io/netfix/demo/frag_bunny.mp4"
}, {
from: 0,
to: 5,
src: "https://raw.githubusercontent.com/w3c/web-platform-tests/master/media-source/mp4/test.mp4"
}, {
from: 0,
to: 5,
src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerBlazes.mp4"
}, {
from: 0,
to: 5,
src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerJoyrides.mp4"
}, {
src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerMeltdowns.mp4",
from: 0,
to: 6
}];
let duration = urls.reduce((n, {
from, to
}) => n + (to - from), 0);
let offlineAudioContext = new OfflineAudioContext(2, 44100 * 60, 44100);
offlineAudioContext.onstatechange = e => console.log(e, e.target.state);
let audioBuffers = await Promise.all(
urls.map(({
src
}) => fetch(src).then(async(response) => audioContext.decodeAudioData(await response.arrayBuffer())))
);
let sources = audioBuffers.map(ab => {
let source = offlineAudioContext.createBufferSource();
source.buffer = ab;
source.connect(offlineAudioContext.destination);
return source;
});
let started = false;
sources.reduce((promise, source, index) => {
return promise.then(() => new Promise(resolve => {
source.start(0, urls[index].from, urls[index].to);
if (started) {
offlineAudioContext.resume();
}
source.onended = () => {
resolve();
if (!started) {
started = true;
offlineAudioContext.suspend(offlineAudioContext.currentTime);
}
};
}))}, Promise.resolve());
let rendering = offlineAudioContext.startRendering()
.then(ab => {
let source = audioContext.createBufferSource();
source.buffer = ab;
source.onended = e => console.log(e, source, offlineAudioContext, audioContext);
source.connect(audioContext.destination);
source.start(0);
})
})();
</script>
</body>
</html>
or without using the resume
method, which is not mentioned in the current specification
<!DOCTYPE html>
<html>
<head>
</head>
<body>
<script>
(async() => {
let audioContext = new AudioContext();
let urls = [{
src: "https://upload.wikimedia.org/wikipedia/commons/a/a4/Xacti-AC8EX-Sample_video-001.ogv",
from: 0,
to: 4
}, {
src: "https://mirrors.creativecommons.org/movingimages/webm/ScienceCommonsJesseDylan_240p.webm",
from: 10,
to: 15
}, {
from: 55,
to: 60,
src: "https://nickdesaulniers.github.io/netfix/demo/frag_bunny.mp4"
}, {
from: 0,
to: 5,
src: "https://raw.githubusercontent.com/w3c/web-platform-tests/master/media-source/mp4/test.mp4"
}, {
from: 0,
to: 5,
src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerBlazes.mp4"
}, {
from: 0,
to: 5,
src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerJoyrides.mp4"
}, {
src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerMeltdowns.mp4",
from: 0,
to: 6
}];
let duration = urls.reduce((n, {
from, to
}) => n + (to - from), 0);
let offlineAudioContext = new OfflineAudioContext(2, 44100 * duration, 44100);
let audioBuffers = await Promise.all(
urls.map(({
src
}) => fetch(src).then(async(response) => audioContext.decodeAudioData(await response.arrayBuffer())))
);
let sources = audioBuffers.map(ab => {
let source = offlineAudioContext.createBufferSource();
source.buffer = ab;
source.connect(offlineAudioContext.destination);
return source;
});
sources.reduce((promise, source, index) => { return promise.then(() => new Promise(resolve => {
source.start(0, urls[index].from, urls[index].to);
source.onended = resolve;
}))}, Promise.resolve());
let rendering = offlineAudioContext.startRendering()
.then(ab => {
let source = audioContext.createBufferSource();
source.buffer = ab;
source.onended = e => console.log(e, source, offlineAudioContext, audioContext);
source.connect(audioContext.destination);
source.start(0);
})
})();
</script>
</body>
</html>
though the result from the above is inconsistent as to gaps in playback or playback at all (no video or audio track?) and we would be using a decodeMediaData
method to get both audio and video data, perhaps separately, and concatenated into a single buffer.
Finally composed a pattern using "segments"
mode with .timestampOffset
and .abort()
const sourceOpen = e => {
console.log(e);
const sourceBuffer = mediaSource.addSourceBuffer(mimeCodec);
sourceBuffer.addEventListener("updateend", e => {
if (mediaSource.duration === Infinity) {
mediaSource.duration = duration;
}
console.log(e, video.currentTime, mediaSource.sourceBuffers[0].updating);
});
sourceBuffer.appendBuffer(chunks.shift());
}
const handleWaiting = e => {
console.log(e, video.currentTime);
if (chunks.length) {
mediaSource.sourceBuffers[0].abort();
mediaSource.sourceBuffers[0].timestampOffset = video.currentTime;
mediaSource.sourceBuffers[0].appendBuffer(chunks.shift());
} else {
console.log(e);
video.autoplay = false;
video.removeEventListener("waiting", handleWaiting);
}
}
The audio and video are not precisely synchronized, though a start at using the https://plnkr.co/edit/E8OlvwiUmCwIUTOKSNkv at version 5.
Have not yet tried with MediaRecorder
.
@wolenetz
Was able to put together code which records MediaSource
at Firefox 60
<!DOCTYPE html>
<html>
<head>
<title>Record media fragments of any MIME type to single video using HTMLMediaElement.captureStream(), MediaRecorder, and MediaSource</title>
</head>
<body>
<script>
const captureStream = mediaElement =>
!!mediaElement.mozCaptureStream ? mediaElement.mozCaptureStream() : mediaElement.captureStream();
class MediaFragmentRecorder {
constructor({
urls = [], video = document.createElement("video"), width = 320, height = 280
} = {}) {
if (urls.length === 0) {
throw new TypeError("no urls passed to MediaFragmentRecorder");
}
return (async() => {
video.height = height;
video.width = width;
video.autoplay = true;
video.preload = "auto";
video.controls = true;
// video.crossOrigin = "anonymous";
const chunks = [];
let duration = 0;
let media = await Promise.all(
urls.map(async({
from, to, src
}, index) => {
const url = new URL(src);
// get media fragment hash from `src`
if (url.hash.length) {
[from, to] = url.hash.match(/\d+/g);
}
return {
blob: await fetch(src).then(response => response.blob()),
from,
to
}
}));
// using `Promise.all()` here apparently is not the same
// as recording the media in sequence as the next video
// is not played when a buffer is added to `MediaSource`
for (let {
from, to, blob
}
of media) {
await new Promise(async(resolve) => {
let recorder;
const blobURL = URL.createObjectURL(blob);
video.addEventListener("playing", e => {
const mediaStream = captureStream(video);
recorder = new MediaRecorder(mediaStream, {
mimeType: "video/webm;codecs=vp8,opus"
});
recorder.start();
recorder.addEventListener("stop", e => {
resolve();
console.log(e);
}, {
once: true
});
recorder.addEventListener("dataavailable", async(e) => {
console.log(e);
chunks.push(await new Response(e.data).arrayBuffer());
URL.revokeObjectURL(blobURL);
});
video.addEventListener("pause", e => {
if (recorder.state === "recording") {
recorder.stop();
} else {
recorder.requestData();
}
console.log(video.played.end(0) - video.played.start(0), video.currentTime - from, video.currentTime);
duration += video.currentTime - from;
}, {
once: true
});
}, {
once: true
});
video.addEventListener("canplay", e => video.play(), {
once: true
});
video.src = `${blobURL}#t=${from},${to}`;
})
};
// using same `<video>` element at `.then()` does not
// return expected result as to `autoplay`
video.load();
return {
chunks, duration, width, height, video
}
})()
}
}
let urls = [{
src: "https://upload.wikimedia.org/wikipedia/commons/a/a4/Xacti-AC8EX-Sample_video-001.ogv",
from: 0,
to: 4
}, {
src: "https://mirrors.creativecommons.org/movingimages/webm/ScienceCommonsJesseDylan_240p.webm#t=10,20"
}, {
from: 55,
to: 60,
src: "https://nickdesaulniers.github.io/netfix/demo/frag_bunny.mp4"
}, {
from: 0,
to: 5,
src: "https://raw.githubusercontent.com/w3c/web-platform-tests/master/media-source/mp4/test.mp4"
}, {
from: 0,
to: 5,
src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerBlazes.mp4"
}, {
from: 0,
to: 5,
src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerJoyrides.mp4"
}, {
src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerMeltdowns.mp4#t=0,6"
}];
new MediaFragmentRecorder({
urls
})
.then(({
chunks, duration, width, height, video
}) => {
let recorder, mediaStream;
console.log(chunks, duration);
document.body.appendChild(video);
const mediaSource = new MediaSource();
const mimeCodec = "video/webm;codecs=vp8,opus";
const sourceOpen = e => {
console.log(e);
const sourceBuffer = mediaSource.addSourceBuffer(mimeCodec);
sourceBuffer.addEventListener("updateend", e => {
console.log(e, video.currentTime, mediaSource);
});
sourceBuffer.appendBuffer(chunks.shift());
}
const handleWaiting = e => {
console.log(e, video.currentTime, recorder && recorder.state);
if (chunks.length) {
// mediaSource.sourceBuffers[0].abort();
mediaSource.sourceBuffers[0].timestampOffset = video.currentTime;
mediaSource.sourceBuffers[0].appendBuffer(chunks.shift());
} else {
video.removeEventListener("waiting", handleWaiting);
mediaSource.duration = video.currentTime;
mediaSource.endOfStream();
console.log(video.currentTime, duration, mediaSource.duration);
try {
if (recorder.state === "recording") {
recorder.stop();
video.removeEventListener("waiting", handleWaiting);
mediaSource.duration = video.currentTime;
mediaSource.endOfStream();
console.log(video.currentTime, duration, mediaSource.duration);
}
} catch (e) {
console.error(e.stack);
console.trace();
}
}
}
mediaSource.sourceBuffers.addEventListener("addsourcebuffer", e => console.log(e));
video.addEventListener("canplay", e => {
console.log(video.readyState, video.paused);
if (video.paused) {
video.play();
}
console.log(e, duration, video.buffered.end(0), video.seekable.end(0), video.duration, mediaSource.duration);
});
video.addEventListener("playing", e => {
mediaStream = captureStream(video);
mediaStream.addEventListener("inactive", e => console.log(e));
recorder = new MediaRecorder(mediaStream, {
mimeType: "video/webm;codecs=vp8,opus"
});
recorder.addEventListener("dataavailable", e => {
let media = document.createElement("video");
media.width = width;
media.height = height;
media.controls = true;
document.body.appendChild(media);
media.src = URL.createObjectURL(e.data);
});
recorder.addEventListener("stop", e => console.log(e));
recorder.start();
console.log(e, duration, video.buffered.end(0), video.seekable.end(0), video.duration, mediaSource.duration);
}, {
once: true
});
video.addEventListener("waiting", handleWaiting);
video.addEventListener("pause", e => console.log(e));
video.addEventListener("stalled", e => console.log(e));
video.addEventListener("loadedmetadata", e => console.log(e));
video.addEventListener("loadeddata", e => console.log(e));
video.addEventListener("seeking", e => console.log(e));
video.addEventListener("seeked", e => console.log(e));
video.addEventListener("durationchange", e => console.log(e));
video.addEventListener("abort", e => console.log(e));
video.addEventListener("emptied", e => console.log(e));
video.addEventListener("suspend", e => console.log(e));
mediaSource.addEventListener("sourceopen", sourceOpen);
video.src = URL.createObjectURL(mediaSource);
})
</script>
</body>
</html>
Observations:
The implementations of MediaSource
at Chromium and Firefox are different in several ways.
Firefox 60 using "segments"
mode
mozCaptureStream()
;
b. when ArrayBuffer
representations of recorded Blob
s of media fragments are passed to .appendBuffer()
;<video>
with MediaSource
set at src
;.abort()
does not need to be executed before setting .timestampOffset
autoplay
set as either a property video.autoplay = true
or video.setAttribute("autoplay", true)
has no effect; the <video>
does not begin playback when the attribute is set, canplay
event needs to be used to execute video.play()
, although autoplay
does begin playback when using a src
other than the recorded Blob
converted to ArrayBuffer
<!DOCTYPE html>
<html>
Have been attempting to implement, for lack of a more descriptive reference, an "offline media context". The basic concept is to be able to use the tools available at the most recent browsers alone to record or request media fragments capable of independent playback and to be able to concatenate those discrete media fragments into a single stream of media playback at an
HTMLMediaElement
. A brief summary of the progression of the proof of concept Proposal: Implement OfflineMediaContext #2824.From the outset have had the sense that
MediaSource
could possibly be utilized to achieve part if not all of the requirement. However, had not located an existing or configured an appropriate pattern during own testing to realize the concatenation of discrete files usingMediaSource
.Found this question and answer How do i append two video files data to a source buffer using media source api? which appeared to indicate that setting the
.timestampOffset
property ofMediaSource
could result in sequencing media playback of discrete buffers appended toSourceBuffer
. Following the question led to a 2012 Editor's Draft Media Source Extensions W3C Editor's Draft 8 October 2012 which states at 2.11. Applying Timestamp OffsetsWhich tried dozens of times using different patterns over the past several days. Interestingly all attempts using
.webm
video files failed; generally resulting in the following being logged atconsole
at plnkrAll of attempts using
.mp4
video files failed save for a single.mp4
file which is a downloaded copy of "Big Buck Bunny" trailer. Not entirely sure where downloaded the file from during testing, though may have been "https://nickdesaulniers.github.io/netfix/demo/frag_bunny.mp4". Is this fact related to what is revealed at FFmpeg FAQ?
Made a copy of the original file in the same directory. Used
<input type="file">
withmultiple
attribute set to upload the files. Converted theFile
objects toArrayBuffer
usingFileReader
and used, in pertinent part, this patternThe questions that have for authors and contributors to the specification are
What is the correct code pattern (as clear and definitive as possible) to use to append array buffers from discrete files or media fragments to one or more
SourceBuffer
s ofMediaSource
, where theHTMLMediaElement
renders playback of each of the files or media fragments?Why was a single
.mp4
which was copied the only two files whichMediaSource
correctly set the.duration
of the to total time of the two files and rendered playback?