search

Ichunjo / vardautomation

Encoding automation framework
https://vardautomation.readthedocs.io/
19 stars 2 forks source link

mkvmerge error after writing file #106

Closed LightArrowsEXE closed 2 years ago

LightArrowsEXE commented 2 years ago

https://discord.com/channels/@me/599711298275770398/972537849524154378

Some unknown error can occur when running mkvmerge that can't be caught in a try/except clause. The final output is still being muxed, but it seems to throw an error when trying to exit, taking the entire process down with it.

Removing check=True in tooling/base.py:44 as done here seems to resolve the error

-            subprocess.run(self.params, check=True, text=True, encoding='utf-8')
+            subprocess.run(self.params, text=True, encoding='utf-8')

This might be a temporary solution for the time being, but a more permanent solution should be found asap.

Ichunjo commented 2 years ago

Can you test with this snippet?

from vardautomation import (
    JAPANESE, X264, X265, Eac3toAudioExtracter, FileInfo2, FlacEncoder, MatroskaFile, MediaTrack,
    PresetAAC, PresetBD, PresetFLAC, QAACEncoder, RunnerConfig, SelfRunner, SoxCutter
)

file = FileInfo2('', (24, 1000), preset=[PresetBD, PresetFLAC])
assert file.a_enc_cut

config = RunnerConfig(
    # X264(['-o', '{clip_output:s}', '-', '--demuxer', 'y4m', '--preset', 'ultrafast']),
    X265(['-o', '{clip_output:s}', '-', '--y4m', '--preset', 'veryfast', '--range', 'limited']),
    None,
    Eac3toAudioExtracter(file, track_in=2, track_out=1, eac3to_args=['-log=nul']),
    SoxCutter(file, track=1),
    # QAACEncoder(file, track=1),
    FlacEncoder(file, track=1),
    MatroskaFile(
        file.name_file_final,
        [MediaTrack(file.name_clip_output, '', JAPANESE),
         MediaTrack(file.a_enc_cut.set_track(1), '', JAPANESE)],
        '--ui-language', 'en'
    )
)

# file.clip_cut = file.clip_cut.std.SetFrameProps(_)

runner = SelfRunner(file.clip_cut, file, config)
runner.run()
runner.work_files.clear()
LightArrowsEXE commented 2 years ago

Can you test with this snippet?

Runs without issue.

$ python UBWBD_NCOP2_test.py 

+––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––+
|                        _                _                            _    _                |
|                       | |              | |                          | |  (_)               |
| __   __ __ _  _ __  __| |  __ _  _   _ | |_  ___   _ __ ___    __ _ | |_  _   ___   _ __   |
| \ \ / // _` || '__|/ _` | / _` || | | || __|/ _ \ | '_ ` _ \  / _` || __|| | / _ \ | '_ \  |
|  \ V /| (_| || |  | (_| || (_| || |_| || |_| (_) || | | | | || (_| || |_ | || (_) || | | | |
|   \_/  \__,_||_|   \__,_| \__,_| \__,_| \__|\___/ |_| |_| |_| \__,_| \__||_| \___/ |_| |_| |
|                                                                                            |
+––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––+

SelfRunning...
X265 command: x265 -o E:\Anime\Fansubbing\Git Repositories\Encoding Projects\[Kaleido-subs]\Work in Progress\Fate stay night - Unlimited Blade Works [BD]_hidden\UBWBD_NCOP2_test - --y4m --preset veryfast --range limited
y4m  [info]: 1920x1080 fps 24000/1001 i420p10 unknown frame count
raw  [info]: output file: E:\Anime\Fansubbing\Git Repositories\Encoding Projects\[Kaleido-subs]\Work in Progress\Fate stay night - Unlimited Blade Works [BD]_hidden\UBWBD_NCOP2_test
x265 [info]: HEVC encoder version 3.5+8-57e817329
x265 [info]: build info [Windows][MSVC 1900][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: Main profile, Level-4 (Main tier)
x265 [info]: Thread pool created using 12 threads
x265 [info]: Slices                              : 1
x265 [info]: frame threads / pool features       : 3 / wpp(17 rows)
VapourSynth: 4/976 ~ 0.41% || Encoder: x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge         : hex / 57 / 1 / 2
x265 [info]: Keyframe min / max / scenecut / bias  : 23 / 250 / 40 / 5.00
x265 [info]: Lookahead / bframes / badapt        : 15 / 4 / 0
x265 [info]: b-pyramid / weightp / weightb       : 1 / 1 / 0
x265 [info]: References / ref-limit  cu / depth  : 2 / on / on
x265 [info]: AQ: mode / str / qg-size / cu-tree  : 2 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress            : CRF-28.0 / 0.60
x265 [info]: tools: rd=2 psy-rd=2.00 early-skip rskip mode=1 signhide tmvp
x265 [info]: tools: fast-intra strong-intra-smoothing lslices=6 deblock sao
x265 [info]: frame I:     18, Avg QP:27.16  kb/s: 5951.98 
x265 [info]: frame P:    202, Avg QP:28.80  kb/s: 1928.76
x265 [info]: frame B:    756, Avg QP:34.20  kb/s: 545.68
x265 [info]: Weighted P-Frames: Y:4.0% UV:2.0%
x265 [info]: consecutive B-frames: 10.5% 3.2% 1.4% 2.3% 82.7%

encoded 976 frames in 40.80s (23.92 fps), 931.64 kb/s, Avg QP:32.95
Writing a_src_cut to E:\Anime\Fansubbing\Git Repositories\Encoding Projects[Kaleido-subs]\Work in Progress\Fate stay night - Unlimited Blade Works [BD]_hidden\UBWBD_NCOP2 (Premux)_cut_track_1.wav ----- 637/637 100.00% 3697.… 0:00:… 
                                                                                                                                                                                                                          fps
ffmpeg command: ffmpeg -i E:\Anime\Fansubbing\Git Repositories\Encoding Projects\[Kaleido-subs]\Work in Progress\Fate stay night - Unlimited Blade Works [BD]_hidden\UBWBD_NCOP2 (Premux)_cut_track_1.wav -compression_level 12 -lpc_type cholesky -lpc_passes 3 -exact_rice_parameters 1 E:\Anime\Fansubbing\Git Repositories\Encoding Projects\[Kaleido-subs]\Work in Progress\Fate stay night - Unlimited Blade Works [BD]_hidden\UBWBD_NCOP2 (Premux)_cut_enc_track_1.flac  
ffmpeg version 2021-10-18-git-d04c005021-full_build-www.gyan.dev Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 10.3.0 (Rev5, Built by MSYS2 project)
  configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-lzma --enable-libsnappy --enable-zlib --enable-librist --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-libbluray --enable-libcaca --enable-sdl2 --enable-libdav1d --enable-libzvbi --enable-librav1e --enable-libsvtav1 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-libaom --enable-libopenjpeg --enable-libvpx --enable-libass --enable-frei0r --enable-libfreetype --enable-libfribidi --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-ffnvcodec --enable-nvdec --enable-nvenc --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-libglslang --enable-vulkan --enable-opencl --enable-libcdio --enable-libgme --enable-libmodplug --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libshine --enable-libtheora --enable-libtwolame --enable-libvo-amrwbenc --enable-libilbc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-ladspa --enable-libbs2b --enable-libflite --enable-libmysofa --enable-librubberband --enable-libsoxr --enable-chromaprint
  libavutil      57.  7.100 / 57.  7.100
  libavcodec     59. 12.100 / 59. 12.100
  libavformat    59.  6.100 / 59.  6.100
  libavdevice    59.  0.101 / 59.  0.101
  libavfilter     8. 14.100 /  8. 14.100
  libswscale      6.  1.100 /  6.  1.100
  libswresample   4.  0.100 /  4.  0.100
  libpostproc    56.  0.100 / 56.  0.100
[w64 @ 000001aab371da80] Estimating duration from bitrate, this may be inaccurate
Input #0, w64, from 'E:\Anime\Fansubbing\Git Repositories\Encoding Projects\[Kaleido-subs]\Work in Progress\Fate stay night - Unlimited Blade Works [BD]_hidden\UBWBD_NCOP2 (Premux)_cut_track_1.wav':
  Duration: 00:00:40.71, bitrate: 3072 kb/s
  Stream #0:0: Audio: pcm_f32le ([3][0][0][0] / 0x0003), 48000 Hz, stereo, flt, 3072 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (pcm_f32le (native) -> flac (native))
Press [q] to stop, [?] for help
[flac @ 000001aab3773c40] encoding as 24 bits-per-sample
Output #0, flac, to 'E:\Anime\Fansubbing\Git Repositories\Encoding Projects\[Kaleido-subs]\Work in Progress\Fate stay night - Unlimited Blade Works [BD]_hidden\UBWBD_NCOP2 (Premux)_cut_enc_track_1.flac':
  Metadata:
    encoder         : Lavf59.6.100
  Stream #0:0: Audio: flac, 48000 Hz, stereo, s32 (24 bit), 128 kb/s
    Metadata:
      encoder         : Lavc59.12.100 flac
size=    8217kB time=00:00:40.70 bitrate=1653.6kbits/s speed= 6.5x    
video:0kB audio:8209kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.098572%
mkvmerge command: mkvmerge -o UBWBD_NCOP2_test.mkv --ui-language en --language 0:jpn E:\Anime\Fansubbing\Git Repositories\Encoding Projects\[Kaleido-subs]\Work in Progress\Fate stay night - Unlimited Blade Works [BD]_hidden\UBWBD_NCOP2_test --language 0:jpn E:\Anime\Fansubbing\Git Repositories\Encoding Projects\[Kaleido-subs]\Work in Progress\Fate stay night - Unlimited Blade Works [BD]_hidden\UBWBD_NCOP2 (Premux)_cut_enc_track_1.flac
mkvmerge v67.0.0 ('Under Stars') 64-bit
'E:\Anime\Fansubbing\Git Repositories\Encoding Projects\[Kaleido-subs]\Work in Progress\Fate stay night - Unlimited Blade Works [BD]_hidden\UBWBD_NCOP2_test': Using the demultiplexer for the format 'HEVC/H.265'.
+-> Parsing the FLAC file. This can take a LONG time.
+-> Pre-parsing FLAC file: 100%
'E:\Anime\Fansubbing\Git Repositories\Encoding Projects\[Kaleido-subs]\Work in Progress\Fate stay night - Unlimited Blade Works [BD]_hidden\UBWBD_NCOP2 (Premux)_cut_enc_track_1.flac': Using the demultiplexer for the format 'FLAC'.  
'E:\Anime\Fansubbing\Git Repositories\Encoding Projects\[Kaleido-subs]\Work in Progress\Fate stay night - Unlimited Blade Works [BD]_hidden\UBWBD_NCOP2_test' track 0: Using the output module for the format 'HEVC/H.265 (unframed)'.  
'E:\Anime\Fansubbing\Git Repositories\Encoding Projects\[Kaleido-subs]\Work in Progress\Fate stay night - Unlimited Blade Works [BD]_hidden\UBWBD_NCOP2 (Premux)_cut_enc_track_1.flac' track 0: Using the output module for the format 'FLAC'.
The file 'UBWBD_NCOP2_test.mkv' has been opened for writing.
Progress: 100%
The cue entries (the index) are being written...
Multiplexing took 0 seconds.
Ichunjo commented 2 years ago

After another tests, we decided to close the issue since we can’t reproduce this bug again. Feel free to open the issue again if you run into it again.

begna112 commented 2 years ago

I've had the same issue. 

encoded 34046 frames, 2.34 fps, 2011.94 kb/s
Resumable encode; merging...
mkvmerge command: mkvmerge -o L:\encode\Utawarerumono - Futari no Hakuoro\utaware_01_part_000.mkv --quiet L:\encode\Utawarerumono - Futari no Hakuoro\utaware_01_part_000
mkvextract command: mkvextract L:\encode\Utawarerumono - Futari no Hakuoro\utaware_01_tmp.mkv tracks 0:L:\encode\Utawarerumono - Futari no Hakuoro\utaware_01
Extracting track 0 with the CodecID 'V_MPEG4/ISO/AVC' to the file 'L:\encode\Utawarerumono - Futari no Hakuoro\utaware_01'. Container format: AVC/H.264 elementary stream
Progress: 100%
mkvmerge command: mkvmerge -o L:\encode\Utawarerumono - Futari no Hakuoro\Premux\utaware_01 (Premux).mkv --ui-language en --language 0:jpn L:\encode\Utawarerumono - Futari no Hakuoro\utaware_01 --language 1:jpn --track-name 1:AAC 2.0 --tags 0:_settings/tags_aac.xml L:\encode\Utawarerumono - Futari no Hakuoro\utaware_01_2_cut.aac
mkvmerge v65.0.0 ('Too Much') 64-bit
'L:\encode\Utawarerumono - Futari no Hakuoro\utaware_01': Using the demultiplexer for the format 'AVC/H.264'.
'L:\encode\Utawarerumono - Futari no Hakuoro\utaware_01_2_cut.aac': Using the demultiplexer for the format 'AAC'.
'L:\encode\Utawarerumono - Futari no Hakuoro\utaware_01' track 0: Using the output module for the format 'AVC/H.264 (unframed)'.
'L:\encode\Utawarerumono - Futari no Hakuoro\utaware_01_2_cut.aac' track 0: Using the output module for the format 'AAC'.
Warning: 'L:\encode\Utawarerumono - Futari no Hakuoro\utaware_01_2_cut.aac': A track with the ID 1 was requested but not found in the file. The corresponding option will be ignored.
The file 'L:\encode\Utawarerumono - Futari no Hakuoro\Premux\utaware_01 (Premux).mkv' has been opened for writing.
Progress: 100%
The cue entries (the index) are being written...
Multiplexing took 1 second.
An error has been caught in function '_do_tooling', process 'MainProcess' (3956), thread 'MainThread' (22696):
Traceback (most recent call last):
  File "L:\encode\Utawarerumono - Futari no Hakuoro\utaware_01.py", line 86, in <module>
    runner.run()
  File "C:\Python310\lib\site-packages\vsencode\encoder.py", line 135, in run
    runner.run()
  File "C:\Python310\lib\site-packages\vardautomation\automation.py", line 157, in run
    f()
  File "C:\Python310\lib\site-packages\vardautomation\automation.py", line 251, in _mux
    wf = self.config.mkv.mux(True)
  File "C:\Python310\lib\site-packages\vardautomation\tooling\mux.py", line 326, in mux
    BasicTool(BinaryPath.mkvmerge, self.command).run()
  File "C:\Python310\lib\site-packages\vardautomation\tooling\base.py", line 35, in run
    self._do_tooling()
> File "C:\Python310\lib\site-packages\vardautomation\tooling\base.py", line 45, in _do_tooling
    subprocess.run(self.params, check=True, text=True, encoding='utf-8')
  File "C:\Python310\lib\subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['mkvmerge', '-o', 'L:\\encode\\Utawarerumono - Futari no Hakuoro\\Premux\\utaware_01 (Premux).mkv', '--ui-language', 'en', '--language', '0:jpn', 'L:\\encode\\Utawarerumono - Futari no Hakuoro\\utaware_01', '--language', '1:jpn', '--track-name', '1:AAC 2.0', '--tags', '0:_settings/tags_aac.xml', 'L:\\encode\\Utawarerumono - Futari no Hakuoro\\utaware_01_2_cut.aac']' returned non-zero exit status 1.
Ichunjo commented 2 years ago

The warning is raised by mkvmerge because a tid was specified in the Track constructor. Since utaware_01_2_cut.aac isn't a matroska file it doesn't have a TID. Easy fix is to just to not specify the TID. The list order of the tracks is enough to keep the same order in the final mkv file