Planning to add a demuxer anytime soon?

[pabloruizetc]

Just wondering if there are any plans to throw in a demultiplexer feature into the library at some point. Got anything like that on the roadmap?

[Vanilagy]

It is on my mind, yes, but it would not be thrown into this library but instead a new one. I wonder what the demand for a demuxer is! I haven't needed it personally, but my guess would be that VideoDecoder is completely useless without one.

What do you need one for?

Demuxing comes with its own set of challenges, since suddenly you're dealing with arbitrary input. Different MP4 files, creates by different muxers and formatted differently, across different versions. Let alone malicious files that are crafted to crash software! So, definitely not trivial, even though I could probably throw together a naive MP4 parser in a weekend. I'm very familiar with the format now :)

[pabloruizetc]

When encoding a video, you must set the currentTime for each frame individually, which can be a slow process. After setting the time, you then need to create a VideoFrame. Alternatively, using a VideoDecoder can streamline this process by decoding all the frames, which is significantly faster. However, as you mentioned, a VideoDecoder is completely useless without a demuxer.

[Vanilagy]

Not quite sure what you mean. By which process are you getting your VideoFrames?

[pabloruizetc]

Consider a scenario where I aim to create a video in the browser, incorporating a text overlay onto an MP4 video file. Currently, my process involves using a HTMLVideoElement to manage each frame. For every frame I want to encode, I set the currentTime to the HTMLVideoElement, wait for the video to seek to that frame, draw it onto a HTMLCanvasElement, add the text overlay to the canvas, and then create a VideoFrame from the canvas to proceed with encoding. The main bottleneck here is setting the currentTime, which is notably slow.

A more efficient approach would involve demuxing the MP4 file to directly access and decode the frames with the VideoDecoder. This would allow me to iterate through the frames more quickly (without waiting to be seeked), apply the text overlay on the canvas for each frame, and streamline the entire video encoding process.

[Vanilagy]

Understood. Yes, exactly that would be the use of a demuxer. It's funny because the browser is already demuxing the file when it's playing back as a video element, but that API is not exposed directly but only through hacks like setting currentTime like you are doing. So, totally agree, would be very cool.

[pabloruizetc]

Hopefully, one day the browser will expose the full ffmpeg library :)

[Vanilagy]

Not sure they use ffmpeg for demuxing!

[ivanjaros]

Hi, i need to add authentication header to a GET request for a standard video file url. Which means i have no other way but to use MediaSource. Can this library support this? Essentially standard behavior as the browser but adds the ability to provide my own url and options to the fetch() method. This does not require demuxing, only muxing...i think. Can this library work in this case?

[Vanilagy]

@ivanjaros I'm not sure what you mean. This library has nothing to do with HTTP!

[Vanilagy]

Closing this issue for now, if I ever make a demuxer I'll make sure to drop an update.