Add method MediaRecorder.restart considering this use case.

[titoBouzout]

Thanks for reading. I have been struggling to solve something very simple that I can't believe wasn't considered.

I have a MediaStream which is recorded via MediaRecorder and that stream is sent to N clients. This works in a perfect world.

In the real world clients just join and leave all the time middle stream. It looks like there's no way for a client to join a stream without restarting the stream, which will of course generate a glitch for every client of the mentioned stream everytime someone joins.

I will try to explain with an example, for the sake of simplicity I left out irrelevant code.

streamer.js -- So this page gets a stream and records it. This page never dies, the stream is infinite.

// streamer.js

var recorder = new MediaRecorder(stream)
recorder.ondataavailable = function(e) {
    if (e.data.size > 0) {
        send_data_to_clients_as_arraybuffer(e.data)
    }
}
recorder.start(1000)

client.js -- So this page receives the blobs from streamer.js and plays it. There could be as many client.js as you need.

// client.js

var media_source = new MediaSource()
media_source.onsourceopen = function (event) {
    var source_buffer = media_source.addSourceBuffer('video/webm;codecs=vp9')
    window.on_data_from_streamer = function(arraybuffer) {
        if (source_buffer.updating === false) {
            source_buffer.appendBuffer(new Uint8Array(arraybuffer))
        }
    }
}
var video = document.createElement('video')
video.src = URL.createObjectURL(media_source)

This code just works in a perfect world. So what's the problem with it?

• a client.js can't just join if the streamer.js already started

Why? Because:

The UA MUST record stream in such a way that the original Tracks can be retrieved at playback time. When multiple Blobs are returned (because of timeslice or requestData()), the individual Blobs need not be playable, but the combination of all the Blobs from a completed recording MUST be playable.

You can't just feed source_buffer.appendBuffer with blobs from the middle of a stream, it will not work.

I know I could just start a new parallel stream.js to serve the client that joined, but my bandwidth will not scale, I can't afford it. I also know about the canvas hack, I need a real solution.

Something I tried:

1. To start and stop the Stream in an interval to generate small videos.

Problems with this:

1. ondataavailable gives you chunks that may not contain a complete last segment. So when you append this video via source_buffer.appendBuffer the buffer gets stuck and will not allow you to update the timestampOffset because is waiting for the remaining data to complete the segment parsing. What this means? You made a small video on which the last segment is incomplete. source_buffer.abort() will just drop that segment which means that you will be dropping segments each time you append a video, so the video and audio will glitch. Im not sure about the following, but as you have an incomplete video, when you create it to get the media.duration, the media duration value that you will pass to timestampOffset could be wrong, making the glitch even more noticeable, but idk. It will depend on, if media.duration considers the time of the last incomplete segment.

2. Restarting the stream is not easy. you can call record.stop() and then immediately call record.start() but these functions are not synced. This means that the first chunk you get in record.start() is not the continuation of the last chunk received when you called record.stop(). So this means that for each small video that you create you will have a double glitch, first because you drop segments and then because you didn't record anything since the time you called stop till the time you called start.

I am missing something? or this is just not possible? If this is possible please let me know how.

In case this is not possible, then a possible solution is to be able to restart the stream WITHOUT loosing any kind of information.

Trying to explain this as in the specification:

restart(optional unsigned long timeslice)

1. [....all the irrelevant items of the specification...]
2. Fire a blob event named dataavailable at recorder with blob. The blob MUST contain a complete segment and not a partial segment. The event could just slice the blob to use the last known segment(the remaining data MUST not be dropped) or the event could wait for the remaining data to complete a segment.
3. Restart the recording
4. dispatch an onrestart event
5. In case the blob was sliced, then the data of that blob MUST be used whenever dataavailable is dispatched.

This way we will be able to generate small videos that can be feed to source_buffer.appendBuffer without source_buffer complaining that segments are incomplete, forcing us to just drop frames. And also we solve the not recorded time between record.stop and record.start.

---

This stuff is just too technical for me, I have no fear to say nonsense. Could we maybe have a way to be able to feed source_buffer.appendBuffer with data that comes from the middle of the stream and on? Like in saving the metada of the recording and have a way to get the "initialization blobs" that will allow source_buffer.appendBuffer to work.

// streamer.js

var recorder = new MediaRecorder(stream)
recorder.ondataavailable = function(e) {
    if (e.data.size > 0) {
        send_data_to_client_as_arraybuffer(e.data)
    }
}
recorder.getinitializationblobs = function(e) {
    if (e.data.size > 0) {
        return e.data
    }
}
recorder.start(1000)

// client.js

source_buffer.appendBuffer(initializationBlobs)

while(chunks_from_the_middle_of_the_stream_and_on)
source_buffer.appendBuffer(chunks_from_the_middle_of_the_stream_and_on)

This will still have the problem that ondataavailable may returns a blob that has the end segment of a previously sent but incomplete segment. Maybe we should have getinitializationblobs and onsegmentavailable.

This API looks like incomplete, did I understand what's happening correctly?. Please inform me, thank you

disclaimer: I just been reading for days about all this and some things I state here could be just my perception and not the reality of whats going on, on which case I would welcome if you can inform me.

[titoBouzout]

Can someone please point us in the right direction, thanks!

[Pehrsons]

Making stop(); then start(); gapless is not supported by the spec, but there is nothing preventing us from making it supported if we want. I have been toying with the idea of using this method to support changing the track set mid-recording, but haven't gotten around to testing how far away we (Firefox) are implementation-wise.

Making this behavior mandatory would be a fair bit of spec work though. I don't have time to work on this myself, but if someone were to commit to contributing a PR and driving it home (i.e., resolve all feedback and get it merged) it might get some traction.

[utyfua]

I think the problem is when playing these fragments, not the recording. How correctly play second "audio"? I dont found solution for this. I use any methods for do that: sourceBuffer.abort, sourceBuffer.timestampOffset, sourceBuffer.remove. That's all i could remember. Example in one page. Sandbox: https://codesandbox.io/s/record-play-case-v7080

index.html

<audio id="audio" controls autoplay />
<div id="status">first fragment(10 sec)</div>
<script src="index.js"></script>

index.js

navigator.mediaDevices.getUserMedia({
    audio: true,
}).then(streamHandle);

var stream, recorder;

function streamHandle(_) {
    stream = _;
    recorderLifetime();
}

function recorderLifetime() {
    recorder = new MediaRecorder(stream);
    recorder.ondataavailable = async event => {
        let data = await blobToArray(event.data);
        sendToAnotherClient(data, event.data.type);
    };
    recorder.start(100);
    setTimeout(() => {
        status.innerHTML = 'all works now?';
        recorder.stop();
        recorderLifetime();
    }, 10000);
};

//another client
var cache = []; // minicache
var mediaSource, sourceBuffer;

function sendToAnotherClient(data, recType) {
    // init playing if not already created
    if (!mediaSource) {
        mediaSource = new MediaSource();
        // audio.srcObject = mediaSource; // dont work in chrome!
        audio.src = URL.createObjectURL(mediaSource);
        mediaSource.addEventListener('sourceopen', () => {
            sourceBuffer = mediaSource.addSourceBuffer(recType);
            // when ready, append all cache to sourceBuffer
            sendToAnotherClient([]);
            audio.play();
        });
    };
    // mediaSource dont ready
    if (!sourceBuffer) {
        cache.push(...data);
        return;
    };
    if (cache.length) {
        data = [...cache, ...data];
        cache = [];
    };
    sourceBuffer.appendBuffer(Uint8Array.from(data));
}

// some support function
function blobToArray(blob) {
    const reader = new FileReader();
    let callback, fallback;
    let promise = new Promise((c, f) => {
        callback = c;
        fallback = f;
    });

    function onLoadEnd(e) {
        reader.removeEventListener('loadend', onLoadEnd, false)
        if (e.error) fallback(e.error);
        else callback([...new Uint8Array(reader.result)]);
    }

    reader.addEventListener('loadend', onLoadEnd, false)
    reader.readAsArrayBuffer(blob);
    return promise;
};

This code works only in chrome, firefoks error:

[Pehrsons]

@utyfua please file a bug with Firefox for this, in the playback component: https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=Audio%2FVideo%3A%20Playback

[utyfua]

@Pehrsons i did it: https://bugzilla.mozilla.org/show_bug.cgi?id=1604419

[jyavenard]

Few issues in your code. First, you must ensure that the format coming out of the recorder is a container supported by MSE (Firefox only supports mp4 and webm). Ogg isn't a supported type with MSE (that's by spec)

Second, your code will call sendToAnotherClient whenever new data is available, which will ultimately call sourceBuffer.appendBuffer(). This will likely error too after the 2nd call to sendToAnotherClient.

Per spec, appendBuffer is an asynchronous operation and the events update, updateend will be fired on the source buffer once the operation is complete. You must wait for that event to have been fired before you can call appendBuffer again, otherwise you will cause an error to be thrown. https://w3c.github.io/media-source/#sourcebuffer-prepare-append "If the updating attribute equals true, then throw an InvalidStateError exception and abort these steps." the updating attribute is set to true when you call appendBuffer and it goes back to being false once the updateend event is fired.

[utyfua]

@Pehrsons thanks for solution! I duplicate your answer here:

Very cool, I got a chance to play with MSE a bit. With mode "sequence" on the SourceBuffer it works as expected with subsequent recordings/chunks. See https://codesandbox.io/s/record-play-case-3r40c -- Start starts a recording, Stop stops it. Restart Stops then Starts in the same task. So if you Stop, the audio element will play what is buffered and then stall, if you again Start it will continue anew. Restart reveals a tiny bit of a glitch but in general it works well.

If we consider my example of the changes that are needed to fix Fixed sandbox: https://codesandbox.io/s/record-play-case-2-q7foo 1) for Firefox: add { mimeType: "audio/webm;codecs=opus" } option for MediaRecorder(line 15 of my example) 2) set sourceBuffer.mode = "sequence"; (line 40)

@titoBouzout This is the solution to this problem. I did not check it for sagging. I can not yet verify the viability of this solution to this issue. Write me about the results of use, whatever they are

[jyavenard]

I doubt you want sequence when dealing with audio + video track.

[utyfua]

@jyavenard i my plans use separated audio and video streams for record and playing. Example code based on webrtc(also used separated audio and video streams): https://github.com/utyfua/webrtc-unified I have not tested it well enough in practice, but in general it is stable and working code for chrome

[jyavenard]

If you use separate video and audio source buffer and use sequence mode, you will get A/V sync issue 100% guarantee.

We are now even discussing of removing sequence mode for such use. Because it's just not practically usable.

[Pehrsons]

Then AIUI we'd need either of:

• A mode on SourceBuffer that treats the audio track as sequence and offsets video by as much as was used for audio
• A mode on MediaRecorder so that Start(); Stop(); Start(); of a recorder doesn't reset the 2nd recording's timestamps to 0.
• Simply never resetting the timestamps to 0 in MediaRecorder, if media element handles this well.

[jyavenard]

Using segment mode and use timestampOffset is probably what you want.

[Pehrsons]

@jyavenard Excellent. Works even better. https://codesandbox.io/s/record-play-case-3r40c updated.