Example showing WebM to MP4 transmuxing

[lucadalli]

When using browser machinery such as getDisplayMedia and MediaRecorder to record a screen, tab, etc, it is not possible to produce MP4 files, only WebM. MP4 is still the container format that is expected by most users/applications and so tansmuxing the WebM to MP4 is commonly handled with ffmpeg.wasm (await ffmpeg.exec(['-i', 'input.webm', '-c', 'copy', 'output.mp4'])).

ffmpeg.wasm is a fantastic tool but it has one major limitation: it keeps the entire input and output files in virtual file system (MEMFS) which is limited to 2GB. It lacks streaming support and with WORKERFS it's possible to stream the input in, but the output is still written to MEMFS which suffers from the 2GB limitation.

For this reason I am trying to use libav.js to transmux my VP9-encoded WebM to MP4 without transcoding it. I have no prior experience dealing with ffmpeg and its libav libraries and have been toiling away for the past two days trying to figure this out. I have managed to demux the WebM but can't for the life of me figure out the muxing to MP4. All samples I have found in the libav.js, libavjs-webcodecs-bridge and libavjs-webcodecs-polyfill repos are focused on encoding, decoding or transcoding. I do understand that muxing follows after encoding and hence all these examples do show muxing but I still haven't been able to figure out how to mux without decoding and re-encoding.

Is it possible to add an example showing WebM -> MP4 transmuxing?

Below is the code I have so far for demuxing the WebM file.

  await libav.mkreadaheadfile('input', blob)

  const [fmt_ctx, streams] = await libav.ff_init_demuxer_file('input')

  const configs: (AudioDecoderConfig | VideoDecoderConfig | null)[] =
    await Promise.all(
      streams.map((stream) => {
        if (stream.codec_type === libav.AVMEDIA_TYPE_AUDIO)
          return audioStreamToConfig(libav, stream)
        else if (stream.codec_type === libav.AVMEDIA_TYPE_VIDEO)
          return videoStreamToConfig(libav, stream)

        return null
      }),
    )

  // rpkt
  const pkt = await libav.av_packet_alloc()
  while (true) {
    const [result, allPackets] = await libav.ff_read_frame_multi(fmt_ctx, pkt, {
      limit: 1024 * 1024,
    })

    for (let i = 0; i < streams.length; i++) {
      const stream = streams[i]
      const config = configs[i]
      if (!config) continue
      const packets = allPackets[stream.index]
      try {
        if (stream.codec_type === libav.AVMEDIA_TYPE_VIDEO) {
          for (const packet of packets) {
            const evc = packetToEncodedVideoChunk(packet, stream)

            // TODO: pass to muxer
          }
        } else if (stream.codec_type === libav.AVMEDIA_TYPE_AUDIO) {
          for (const packet of packets) {
            const eac = packetToEncodedAudioChunk(packet, stream)

            // TODO: pass to muxer
          }
        }
      } catch (e) {
        console.error(e)
      }
    }

    if (result === libav.AVERROR_EOF) {
      break
    }
  }

  libav.terminate()

  // TODO: finalize muxer

[Yahweasel]

I wish people would stop justifying their use of Misanthropic Patent Extortion Gang garbage to me. I get it. I live in the same world as you do. I just feel that it's important to point out that they're evil and terrible and to avoid their evil and terrible shit when possible.

libav.js supports the CLI, and the files used by the CLI can be devices, just like the files used from the rest of the API. So, why not build the input and output device files, then libav.ffmpeg(exact same arguments you would've given ffmpeg.wasm) will work.

Alternatively, to do it by hand, you'd do exactly what you're doing, and mux in a fairly usual way. The important detail is that ff_init_muxer normally takes codec contexts, but it can take codec parameters with the right flag, and you can get those codec parameters out of the demuxer to pass things through directly. So, take a demuxer test and glue it to a muxer test with that change. I have a bit of a quagmire with all this libav jargon: I didn't write libav, I just wrote a binding of it to JavaScript, so I shouldn't really be the one to document how to do this with libav, just what my wrappers do. But, libav's documentation is craaaaaaaaaaaaaap. So, I can often only say "check the libav documentation", knowing full well that that's a fool's errand unless you've been living neck-deep in libav nonsense for years as I have.

You're correct that there's no demo or example of that right now. Turns out the only test that transmuxes uses the low-level libav API (it's just a conversion of an existing libav test) rather than the high(er)-level libav.js API. libav.js is as complete and as documented as I've made it (or have been paid to make it), and anticipating uses that I don't have is beyond that scope. If you end up writing a transmuxer you're happy with and would like to add it, PRs are welcome. Ultimately, though, transmuxing in this way is probably fairly pointless when you can just use the CLI, as mentioned above :) . Doing it through the CLI will also be faster as less data is moved into and out of JavaScript.

I'm curious... do you find that your Misanthropic Patent Extortion Gang files with VP9 in them actually... work? There are standards for storing non–Misanthropic Patent Extortion Gang codecs in ISOBMF files, but in my experience, the actual support for playing such files is a bit dodgy. Though, my experience is olde, so perhaps that's changed.

[lucadalli]

I hear you and heed your warning but as you're saying, sometimes it's inevitable.

The CLI! Oh man, it was so easy all along. Just so I could figure things out I used the variant-webm-vp9-cli and remuxed WebM back to WebM. If I understand correctly, there is no webcodecs-cli variant so to transmux from WebM to MP4 I will need to create my own that includes fragment cli and the ones that make up variant webcodecs. Given that I am on Windows, something tells me it's gonna be easier said than done.

Oh yeah VP9 MP4 barely play anywhere. Surprisingly even trusty VLC shits the bed but MP4Box.js and Chrome deal with them just fine which is what I need for this particular use case. For user-facing video files I record a WebM with H.264 (eek, sorry) and transmux to MP4.

[Yahweasel]

Yeah, to get a CLI that has (exactly) what you need you'll need to build it; there's no CLI version provided that does exactly that. But, AFAIK, it should work. You can grab webcodecs' config out of configs/configs/webcodes/config.json, and just add CLI to it to make such a config.

[lucadalli]

After many attempts I have managed to build my webcodecs-cli variant on Windows by using Ubuntu on WSL 2. However, when attempting to load my variant I am faced with a ReferenceError: _scriptDir is not defined error. Something seems wrong with the build. When compared to other prebuilt variants I notice that where other variants have variable _scriptDir mine has _scriptName.

WASM Factory file: libav-5.4.6.1.1-webcodecs-cli.dbg.wasm.mjs

[Yahweasel]

What version of Emscripten are you running?

[lucadalli]

luca@LUCA-PC:~/libav.js/emsdk$ emcc -v

emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 3.1.61 (67fa4c16496b157a7fc3377afd69ee0445e8a6e3)
clang version 19.0.0git (https:/github.com/llvm/llvm-project 7cfffe74eeb68fbb3fb9706ac7071f8caeeb6520)
Target: wasm32-unknown-emscripten
Thread model: posix
InstalledDir: /home/luca/libav.js/emsdk/upstream/bin