mkvmerge concat fails on video with large number of chunks on Linux

[shssoichiro]

Received the following error while attempting to concat an encode with 1306 chunks in av1an with the mkvmerge concatenator, on the latest av1an master in Arch Linux x86_64.

ERROR [av1an_core::concat] mkvmerge concatenation failed with output: Output {
    status: ExitStatus(
        unix_wait_status(
            512,
        ),
    ),
    stdout: "mkvmerge v64.0.0 ('Willows') 64-bit\nError: The file '01021.ivf' could not be opened for reading: open file error.\n",
    stderr: "",
}
command: "mkvmerge" "@../options.json"

Cursed.

[redzic]

@shssoichiro Can you see if ulimit -n 20000 fixes the problem?

[shssoichiro]

Yes, that does seem to fix the issue. There's nothing we can do to resolve that from our side though, is there?

[redzic]

The only thing I can think of is to try to check for the error and display an error message that suggests doing ulimit if it happens

[xfzv]

I'm getting the same error:

ERROR [av1an_core::concat] mkvmerge concatenation failed with output: Output {
    status: ExitStatus(
        unix_wait_status(
            512,
        ),
    ),
    stdout: "mkvmerge v67.0.0 ('Under Stars') 64-bit\nError: The file '00000.mkv' could not be opened for reading: open file error.\n",
    stderr: "",
}
command: "mkvmerge" "@../options.json"

Unfortunately, running ulimit -n 20000 beforehand doesn't make any difference.

Any idea?

Thanks.

[Valenciano118]

I had this issue once before, the ulimit didn't work, I had to edit /etc/systemd/system.conf and increase the DefaultLimitNOFILE parameter. I have it set like "DefaultLimitNOFILE=8096:524288".

I don't know if there's a command to apply that config, just reboot after changing it.

Edit: This happened to me on Arch Linux based distros.

[xfzv]

I had this issue once before, the ulimit didn't work, I had to edit /etc/systemd/system.conf and increase the DefaultLimitNOFILE parameter. I have it set like "DefaultLimitNOFILE=8096:524288".

I don't know if there's a command to apply that config, just reboot after changing it.

Edit: This happened to me on Arch Linux based distros.

Thanks for the suggestion but I'm still getting the same error even with DefaultLimitNOFILE=8096:524288 in /etc/systemd/system.conf (I rebooted after modifying the value). I'm using Arch Linux too.

% systemctl --user show | rg LimitNOFILE
LimitNOFILE=524288
LimitNOFILESoft=524288

% sudo systemctl show -p DefaultLimitNOFILE
DefaultLimitNOFILE=524288

However, ulimit -n still returns 1024, not sure if that's normal.

[Valenciano118]

Thanks for the suggestion but I'm still getting the same error even with DefaultLimitNOFILE=8096:524288 in /etc/systemd/system.conf (I rebooted after modifying the value). I'm using Arch Linux too.

% systemctl --user show | rg LimitNOFILE
LimitNOFILE=524288
LimitNOFILESoft=524288

% sudo systemctl show -p DefaultLimitNOFILE
DefaultLimitNOFILE=524288

However, ulimit -n still returns 1024, not sure if that's normal.

Maybe you need to also change DefaultLimitNOFILE on /etc/systemd/user.conf.

Also take a look into limits.conf, but I'm not certain if systemd is above the limits.conf in hierarchy.

There seems to be multiple ways to change the file descriptor limit in Linux so you can check all those other options as well.

[xfzv]

Thanks for the suggestion but I'm still getting the same error even with DefaultLimitNOFILE=8096:524288 in /etc/systemd/system.conf (I rebooted after modifying the value). I'm using Arch Linux too.

% systemctl --user show | rg LimitNOFILE
LimitNOFILE=524288
LimitNOFILESoft=524288

% sudo systemctl show -p DefaultLimitNOFILE
DefaultLimitNOFILE=524288

However, ulimit -n still returns 1024, not sure if that's normal.

Maybe you need to also change DefaultLimitNOFILE on /etc/systemd/user.conf.

Also take a look into limits.conf, but I'm not certain if systemd is above the limits.conf in hierarchy.

There seems to be multiple ways to change the file descriptor limit in Linux so you can check all those other options as well.

Still getting the error after changing DefaultLimitNOFILE value to 8096:524288 in /etc/systemd/user.conf and rebooting.

ulimit -n does return 8096 now that I've edited the user config file, but it doesn't seem to make any difference.

My video has 1508 chunks. I never had this issue with smaller files with less chunks/scenes. I'll have a look at limits.conf, thanks again.

I hope I'll find a solution, I've spent quite a long time encoding this file...

[redzic]

As a workaround you can also try concatenating with ffmpeg if possible, which does not (to my knowledge) attempt to read all the files at the same time which is what causes this issue in the first place.

[xfzv]

As a workaround you can also try concatenating with ffmpeg if possible, which does not (to my knowledge) attempt to read all the files at the same time which is what causes this issue in the first place.

With --concat ffmpeg:

Error: mkvmerge is required for concatenating x265, as x265 outputs raw HEVC bitstream files without the timestamps correctly set, which FFmpeg cannot concatenate properly into a mkv file. Specify mkvmerge as the concatenation method by setting `--concat mkvmerge`.

Could I also try --concat ivf considering I have quite a lot of .ivf files in the encode subdirectory of my av1an temporary directory?

[Valenciano118]

I actually run the command using the root user, have you tried running it as root?

[redzic]

Ah, so you've hit one of the edge cases where using ffmpeg concatenation is not possible (x265 and --enable-keyframe-filtering=2 in aomenc).

Could I also try --concat ivf considering I have quite a lot of .ivf files in the encode subdirectory of my av1an temporary directory?

You could try that as a last resort, but I would not rely on that to be consistently working since there are definitely bugs in the ivf concatenation unfortunately. If you do this do it with --keep so that you can still try other concatenations methods if it doesn't turn out correctly.

Another workaround would be to manually concat with ffmpeg. I know the error message says ffmpeg can't do it, but really what it means is that it can't do it directly to a mkv without first creating an intermediate mp4 (at least, to my knowledge). You could try creating a file that lists all the chunks and then give that to ffmpeg. Here are some more docs on that: https://trac.ffmpeg.org/wiki/Concatenate

[xfzv]

I actually run the command using the root user, have you tried running it as root?

Which command are you talking about?

Ah, so you've hit one of the edge cases where using ffmpeg concatenation is not possible (x265 and --enable-keyframe-filtering=2 in aomenc).

Could I also try --concat ivf considering I have quite a lot of .ivf files in the encode subdirectory of my av1an temporary directory?

You could try that as a last resort, but I would not rely on that to be consistently working since there are definitely bugs in the ivf concatenation unfortunately. If you do this do it with --keep so that you can still try other concatenations methods if it doesn't turn out correctly.

Another workaround would be to manually concat with ffmpeg. I know the error message says ffmpeg can't do it, but really what it means is that it can't do it directly to a mkv without first creating an intermediate mp4 (at least, to my knowledge). You could try creating a file that lists all the chunks and then give that to ffmpeg. Here are some more docs on that: https://trac.ffmpeg.org/wiki/Concatenate

Thanks! it worked:

% cd encode
% printf "file '%s'\n" *.ivf > chunks.txt

% bat -p chunks.txt
file '00000.ivf'
file '00001.ivf'
file '00002.ivf'
file '00003.ivf'
file '00004.ivf'
[...]

% ffmpeg -f concat -i chunks.txt -c copy x265.mp4
% ffmpeg -i x265.mp4 -i audio.mkv x265_final.mkv

Still curious about the mkvmerge error though.

[Valenciano118]

Hi, I'm commenting once again here because I've found a workaround to the mkvmerge stuff though it's still not an ideal solution imo.

I used to run an Arch based distro and by just changing the /etc/systemd/user.conf and /etc/systemd/system.conf with DefaultLimitNOFILE=8192:524288 would be enough to increase the file descriptors and ulimit -n would return a value of 8192.

Recently I changed to Fedora and I just encountered the same issues, but after editing those 2 files, ulimit -n would return 1024 and mkvmerge would fail.

After editing /etc/security/limits.conf and adding these two lines to the configuration:

* hard nofile 65535 * soft nofile 65535

I was able to increase the limit, though ulimit -n would still return 1024. So what I did was to use ulimit -n 4096 on my shell and it successfully increased the limit and mkvmerge would not complain about file descriptors.

I still don't know why the limit doesn't get increased by systemd, but it seems to work well this way. If you wanted it to change on boot you could add it to ~/.bashrc or whatever your shell rc config file is. A bit janky but it worked for me.