Help with multimedia file (muxing audio and video)

[HalexV]

Hi,

I'm with a problem on the last part of transform a mp4 file into a webm file.

I could not understand your explanation on this part of your doc: https://github.com/Vanilagy/webm-muxer#media-chunk-buffering.

I'm trying to pass to muxer the encoded parts of audio and video in a interleaved way, but it is not working.

The image below shows the order that I am sending the encoded chunks to the muxer. The number on front of the label is the chunk timestamp. I tried some orders and the resulted file doesn't play.

Do you have any suggesting about this?

Thank you.

[Vanilagy]

Mh, weird. So, first of all, you don't actually need to do any interleaving yourself manually - the library will automatically interleave the chunks for you internally. The README section you referenced simply states that, since interleaving requires holding chunks in memory for a little while, you should try to avoid buffering too much of one medium when muxing very large/long files to keep memory usage low. All of this is of course irrelevant if you're muxing to an ArrayBufferTarget anyway - it'll be stored in memory anyway. It is relevant if you're writing directly to disk, where you should avoid first muxing 2 gigabytes of video and then 1 gigabyte of audio.

I think something else in your usage of the library must be wrong - could you send me all the relevant code, and maybe a broken output file of you have them?

[HalexV]

The code where I'm using webm-muxer: https://github.com/HalexV/semana-javascript-expert08/blob/challenges/initial-template/app/pages/file-upload/src/worker/videoProcessor.js#L1705

If you want, you can run the project by yourself cloning my repo: https://github.com/HalexV/semana-javascript-expert08/tree/challenges

To run:

cd initial-template/app/

npm start

It will open the browser and you can access: http://localhost:3001/pages/file-upload/

This feature I'm working is an upload of mp4 videos to a server. The mp4 video is decoded and demuxed and while processing, the video is plotted in a canvas. The mp4 video is encoding to 144p and muxed to webm format.

The code is working in progress. Right now a video is selected automatically to be processed and after the video is processed, a blob file is generated in webm format and a download is initiated automatically.

I'm using target StreamTarget and I had to copy and paste your lib code into my code because I couldn't use it any other way.

The broken video file that I generated is in broken-video folder.

[Vanilagy]

Thanks for all this info, I'll check as soon as I find the time :)

[Vanilagy]

Alright, I read a bit through your code. It seems like you're doing some Streams stuff - this is totally fine and works with WebM muxer, but you need to have streaming: true specified in the options to ensure data is written sequentially. Streams operate solely on sequential data, meaning you can't "seek back", which webm-muxer will do by default without streaming: true. I see you have commented out streaming: true in your code - have you tried with it on?

I have inspected your broken file with MKVToolNix, and the file header seems to have been written correctly:

The fact that the rest of the file (all the media chunks) is missing leads me to suspect that the seeking done by .finalize failed because you're working with sequential streams. You therefore need to enable streaming: true.

[Vanilagy]

This also makes me realize that "streaming" is kind of a bad name for this parameter - maybe something like "sequential" would be better? Need to see.

[HalexV]

Yes, I tried streaming: true before open this issue and it worked, but when I play the video I can't see the total duration and I can't jump to anytime on the video and I need this features.

The stream stuff is to process the video on demand and send parts of the video to a server. The server will receive these parts and when finish it will join these parts in a one file.

I'm not an expert on this matter, so if I'm doing something conceptually wrong, I'm sorry.

When you will streaming this file on the server to the client of the user, the player on client needs the info about total duration and if it can jump to anytime on the video?

[Vanilagy]

Yes, you lose seeking and other features when streaming: true is enabled - that's expected, because it writes a simpler version of the file to make sure it does not need to "jump back" to fix old parts of the file. Without streaming: true, you need to make sure to join the chunks emitted by onData correctly! Simply concatenating the buffers, even sorted by position, will not work. You need to write the buffers in the order in which you receive them, making sure to write each buffer to the correct position. So, you do the following:

1. Find the total length s of the file, this is basically Math.max(...chunks.map(x => x.position + x.data.byteLength))
2. Allocate an empty buffer of size s, so something like let buffer = new Uint8Array(s)
3. Iterate over all chunks, in the order in which you received them, and copy the data into buffer at the correct position.

After this is done, you should have a correct file! I assume that's the only thing you've done wrong, concatening the buffers naively instead of writing each buffer at the correct position.

Does this make sense, or do you need help with anything else?

[HalexV]

Yes, It does! Thank you.

But I have some questions:

1. How do I copy the data into buffer at the correct position? It's just buffer[chunk.position] = chunk.data ?
2. The audio chunks and video chunks will have unique positions, so they will be in a possible alternate order, or we can have an audio chunk and a video chunk with the same position in the buffer?

[Vanilagy]

You can copy a buffer into another buffer using TypedArray.prototype.set. Doing buffer[i] = will set just one piece of data, like one byte.

I'm not sure I understand your question regarding audio and video chunks. webm-muxer will properly place these chunks automatically to ensure correct playback :)

[HalexV]

I tried this but it didn't work:

           const fileLength = Math.max(
              ...this.#buffers.map((x) => x.position + x.data.byteLength)
            );
            const resultBuffer = new Uint8Array(fileLength);

            for (const chunk of this.#buffers) {
              // console.log(chunk.data.byteLength, chunk.position);
              resultBuffer.set(chunk.data, chunk.position);
            }

The resulted blob file size should be 21MB but instead it is 53MB and it doesn't play.

I'm feeling that this resultBuffer.set(chunk.data, chunk.position); it's not so simple.

[HalexV]

Now I did it!

The error was here:

Buffers now is a unique Uint8Array and blob's constructor needs to receive an array of other arrays. So I put buffers in one array and it's work! The video is now playable and I can see total duration and playbacks.

Now I will move this logic of create the resultBuffer into webserver and It will create the unique video/webm from the buffers that contain the objects with data and position.

Thank you for your attention!

[Vanilagy]

Perfeito, glad I was able to help! I admit, these operations are a bit more advanced, or better said, require some expertise in buffer processing in JavaScript. But I'm sure you've learned a lot!