search

dalecurtis / mediarecorder-muxer

Explainer for using MediaRecorder to mux WebCodecs generated content.
2 stars 0 forks source link

Re-use MediaStreamTrackGenerator #2

Open chrisguttandin opened 11 months ago

chrisguttandin commented 11 months ago

I was curious if this could be achieved by using a MediaStreamTrackGenerator. I build a little demo for it.

https://stackblitz.com/edit/js-1jfqfv?file=index.js,index.html

It uses a canvas to render red and green rectangles which get then converted into VideoFrames and fed into the writabe of a MediaStreamTrackGenerator. It almost works.

The problem is that the MediaStreamTrackGenerator doesn't seem to respect the duration and the timestamp of a VideoFrame. I'm not sure if this is a bug or if that's intentional. If it would respect that one could generate and "record" a MediaStream using the MediaRecorder at least in real time.

I know that it's already possible to use captureStream() on a canvas to achieve a similar thing. The problem with that solution however is that the timing is unreliable. Being able to craft VideoFrames with a precise duration and timestamp would solve this problem.

Maybe this idea could even be expanded to support faster than realtime rendering.

dalecurtis commented 11 months ago

Interesting demo! I'm surprised duration / timestamp aren't respected though MediaRecorder does a bunch of timestamp rewriting for various reasons, can you file a bug at https://crbug.com/new and I can route to the right folks to take a look?

As you've found MSTG -> MediaRecorder is indeed possible. The only downside is that you can't control encoding parameters or segmentation as precisely as one might like. Maybe that's okay though.