Remuxing from MKV to MP4

[samantehrani]

First of all, thanks for this wrapper and the great initiative.

I have been using LibAV cli mode to remux an MKV to MP4. The process seemingly happens without any errors, however there are bunch of debug outputs complaining about non-monotonous DTS and adjustment of timestamp. Aside from the debug warnings/errors, both Firefox and Chrome will hit a decoding error after few minutes of playback. To give more context:

[MKV Input] -> streaming over wire -> [stream to a block reader device] -> [FFMPEG CLI(command below)] -> [ stream to block writer device] -> [feed to Media Source Extension]

Using latest 5.4.6.1.1 release.

FFMPEG command (for simplicity I am removing the audio from the output):

    "-hide_banner",
    "-loglevel",
    "debug",
    "-nostdin",
    "-i",
    "cfeea6d1-4b1f-4a6a-a6a0-308a536856bd",
    "-an",
    "-c:0",
    "copy",
    "-f",
    "mp4",
    "-sn",
    "-movflags",
    "frag_keyframe+default_base_moof+empty_moov",
    "-y",
    "cbdaf9de-d9de-4eb2-87b1-f14be32ea0ff"
]

The video plays for few minutes (some amount of time, as it varies), before it hits a decoding error both in Firefox and Chrome.

I have been testing a variety of different ffmpeg settings, different MKV inputs, but non to avail.

Ultimately to get a more realistic comparison against using ffmpeg cli directly, I run the same command/process on the identical MKV input:

• on my terminal, store the output. inspect the output.mp4 GPAC:

  PID 1 ID 1 video inMovie duration 01:49:48.789 timescale 16000 1152x496 fps 16000/672 SAR 1/1 2606 kbps 157971 frames codec avc1.640029 AVC|H264 PL High@4.1 YUV 4:2:0 8 bpp

• on browser (using libav.js), store the output. inspect the output.mp4 gpac:

  [iso file] Read Box type traf (0x74726166) at position 43252526 has size 0 but is not at root/file level. Forbidden, skipping end of parent box !
  [iso file] Box "moof" (start 43252502) has 2105093436 extra bytes
  PID 1 ID 1 video inMovie duration 02:44.914 timescale 16000 1152x496 fps 16000 SAR 1/1 2095 kbps 3954 frames codec avc1.640029 AVC|H264 PL High@4.1 YUV 4:2:0 8 bpp

  The browser output is 2.15 Gbs, however it only plays a ~2 minute long video in VLC and as the info shows from gpac.

Here is a some portions of the debug dump from ffmpeg on the browser ( I can figure provide the full dump if you think it is relevant, but these are the bits that stood out to me):

...
[matroska,webm @ 0x12a3980] Unknown entry 0x22B59D at pos. 4386
[matroska,webm @ 0x12a3980] Unknown entry 0x22B59D at pos. 4489
st:0 removing common factor 1000000 from timebase
st:1 removing common factor 1000000 from timebase
...
cur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
frame=    1 fps=0.0 q=-1.0 size=       0kB time=-00:00:00.04 bitrate=N/A speed=N/A    
[mp4 @ 0x12abbf0] Non-monotonous DTS in output stream 0:0; previous: 4672, current: 672; changing to 4673. This may result in incorrect timestamps in the output file.
libav-5.4.6.1.1-lite-av-3.dbg.wasm.js:1985 [mp4 @ 0x12abbf0] Non-monotonous DTS in output stream 0:0; previous: 4673, current: 1328; changing to 4674. This may result in incorrect timestamps in the output file.

I am filing an issue here as I am unsure whether the distinction in the results is being caused by the wrapper, or has to do with something else I am mis-configuring. Would appreciate any thoughts, or possibly a better test case you can suggest to further inspect the cause.

Even though this could be an issue irrelevant the wrapper that you have put around ffmpeg, filing an issue here to rule that possibility out and get your insight.

[Yahweasel]

-f mp4 is by-the-books standard MPEG-4 (not even ISOBMFF), and is not a streamable format. The streamable version is ISMV. I think, but am not certain, that format options can cause the mp4 muxer to behave as a streamable ISMV muxer, and I'm not confident one way or the other on whether your flags will make it do so. At the very least, you should check in the onwrite for the target whether the positions and blocks it's writing actually form a stream. If it's not streaming, it will go back and rewrite MOOF blocks, but if you take that data as streaming, it'll be out of order, so you'll just get corrupted data.

If my intuition is correct, you'll need to use -f ismv. I don't explicitly include ismv in any build of libav.js, but it's possible it gets enabled with the whole MP4 suite. If not, you'll have to make a build with it. (If you do end up needing to do that, I really should add it to the default builds, because it's very useful.)

[samantehrani]

sounds like ismv is not included in the build as I am getting

ismv is not a supported object container type

I will try creating a configuration with support.

[Yahweasel]

From a quick check on the command line, it looks like your movflags actually should be sufficient to make it streamable. But, that doesn't mean it'll actually stream.

To create your output device, are you using mkwriterdev or mkstreamwriterdev? If the former, try the latter. That'll force it to stream the output. (Or, check the position of onwrite events for monotonicity)

[samantehrani]

Changed to using mkstreamwriterdev, same issue. I also made sure that the positions I am getting from the onwrite are monotonous, and they are.

[Yahweasel]

Curiouser and curiouser. It's clearly creating a corrupted MP4 file, but I'm not at all sure why. I'm afraid I don't have an answer for you. You might try using a bigger build (in particular, one with the ability to read or at least parse H.264), and see if that makes any difference. It might be that copying blind is causing it to copy wrong. But that's just a guess, and I'm running low on guesses :)

[samantehrani]

this is the config for the build I am using:

[
    "libvpx",

    "bsf-extract_extradata",
    "bsf-vp9_metadata",
    "bsf-av1_metadata",
    "bsf-h264_metadata",
    "bsf-hevc_metadata",

    "format-mp4",
    "format-aac",
    "format-webm",
    "format-mp3",
    "format-ogg",
    "format-flac",
    "parser-vp8",
    "parser-vp9",

    "parser-hevc",
    "parser-av1",
    "parser-h264",
    "parser-ac3",
    "parser-eac3",
    "parser-aac",
    "parser-opus",
    "parser-flac",

    "decoder-libopenh264",
    "decoder-hevc",
    "decoder-libaom_av1",
    "decoder-ac3",
    "decoder-eac3",
    "decoder-flac",
    "decoder-aac",

    "encoder-libopenh264",
    "encoder-libopus",
    "encoder-libvorbis",
    "encoder-libvpx_vp8",
    "encoder-libvpx_vp9",
    "encoder-aac",

    "cli"
]

[samantehrani]

That's alright, I appreciate that you are giving it a thought. Is there anywhere in the wrapper code that you can point me to start investigating ? Better place to start at I guess?

[Yahweasel]

I'm afraid not. If you're using the CLI, you're almost not using the wrapper code, so there's not much relevant there.

[samantehrani]

Hi, Just some update. I built LibAV with x264 instead of openh264. I no longer get the warnings/errors on non-monotonous DTS. However I am still getting the same result (hitting media decoding error on the browser player). One interesting observation I get from the output of is that fps of the output mp4 is variable and really high (~5000), which doesn't sound correct.

frame= 5740 fps=5708 q=-1.0 size=   63490kB time=00:03:59.28 bitrate=2173.6kbits/s speed= 238x    [NULL @ 0x12aa440] Decoding VUI

setting the framerate -r doesn't help either.