search

slhck / ffmpeg-quality-metrics

Calculate quality metrics with FFmpeg (SSIM, PSNR, VMAF, VIF)
399 stars 52 forks source link

TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType #10

Closed CrypticSignal closed 4 years ago

CrypticSignal commented 4 years ago

I tried running this program with the following command: ffmpeg_quality_metrics -ev ref trim2.mp4 And I got the following error: 

Traceback (most recent call last):
  File "c:\users\h\appdata\local\programs\python\python38\lib\runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\h\appdata\local\programs\python\python38\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\H\AppData\Local\Programs\Python\Python38\Scripts\ffmpeg_quality_metrics.exe\__main__.py", line 7, in <module>
  File "c:\users\h\appdata\local\programs\python\python38\lib\site-packages\ffmpeg_quality_metrics\__main__.py", line 299, in main
    if not os.path.isfile(model_path):
  File "c:\users\h\appdata\local\programs\python\python38\lib\genericpath.py", line 30, in isfile
    st = os.stat(path)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType
CrypticSignal commented 4 years ago

Nevermind, I was using the program incorrectly, You need to specify both a reference file and a "distorted" file

CrypticSignal commented 4 years ago

--enable-vmaf doesn't work for me, though. Here's the command that I used:

ffmpeg_quality_metrics "trim2 [fast].mkv" trim2.mp4 --enable-vmaf

And here's the error that I get:

 C:\Users\H\Desktop\ffmpeg-script>ffmpeg_quality_metrics "trim2 [fast].mkv" trim2.mp4 --enable-vmaf
Traceback (most recent call last):
  File "c:\users\h\appdata\local\programs\python\python38\lib\runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\h\appdata\local\programs\python\python38\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\H\AppData\Local\Programs\Python\Python38\Scripts\ffmpeg_quality_metrics.exe\__main__.py", line 7, in <module>
  File "c:\users\h\appdata\local\programs\python\python38\lib\site-packages\ffmpeg_quality_metrics\__main__.py", line 299, in main
    if not os.path.isfile(model_path):
  File "c:\users\h\appdata\local\programs\python\python38\lib\genericpath.py", line 30, in isfile
    st = os.stat(path)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

Do you know what the issue is?

slhck commented 4 years ago

I think this has to do with this being Windows. I don't know where VMAF installs its models on that platform, and as a consequence, I don't think the code handles that.

How or where did you install VMAF?

CrypticSignal commented 4 years ago

@slhck The requirements section of the README repository only list Python 3.6 and FFmpeg as requirements, and I can see --enable-libvmaf in the configuration of my installation of FFmpeg so it should be fine?

ffmpeg version n4.1.4-ffmpeg-windows-build-helpers Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 8.2.0 (GCC)
  configuration: --pkg-config=pkg-config --pkg-config-flags=--static --extra-version=ffmpeg-windows-build-helpers --enable-version3 --disable-debug --disable-w32threads --arch=x86_64 --target-os=mingw32 --cross-prefix=/home/foxgr/ffmpeg-windows-build-helpers/sandbox/cross_compilers/mingw-w64-x86_64/bin/x86_64-w64-mingw32- --enable-libcaca --enable-gray --enable-libtesseract --enable-fontconfig --enable-gmp --enable-gnutls --enable-libass --enable-libbluray --enable-libbs2b --enable-libflite --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopus --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libzimg --enable-libzvbi --enable-libmysofa --enable-libaom --enable-libopenjpeg --enable-libopenh264 --enable-liblensfun --enable-libvmaf --enable-libsrt --enable-demuxer=dash --enable-libxml2 --enable-nvenc --enable-nvdec --extra-libs=-lharfbuzz --extra-libs=-lm --extra-libs=-lpthread --extra-cflags=-DLIBTWOLAME_STATIC --extra-cflags=-DMODPLUG_STATIC --extra-cflags=-DCACA_STATIC --enable-amf --enable-libmfx --enable-gpl --enable-avisynth --enable-frei0r --enable-filter=frei0r --enable-librubberband --enable-libvidstab --enable-libx264 --enable-libx265 --enable-libxvid --enable-libxavs --enable-avresample --extra-cflags='-mtune=generic' --extra-cflags=-O3 --enable-static --disable-shared --prefix=/home/foxgr/ffmpeg-windows-build-helpers/sandbox/cross_compilers/mingw-w64-x86_64/x86_64-w64-mingw32 --enable-nonfree --enable-decklink --enable-libfdk-aac

I'll try cloning this repository and running it that way.

slhck commented 4 years ago

Yes, in principle this script works for Python and FFmpeg on any platform, except for using it with VMAF — as you've discovered —, which is special because it puts the model files (.pkl) in some location that differs depending on the installation method. I only covered Unix-like systems in this code to auto-discover that location. I didn't include Windows here, and it should have shown you this error message: https://github.com/slhck/ffmpeg-quality-metrics/blob/master/ffmpeg_quality_metrics/__main__.py#L292 (I'll fix the code tomorrow.)

Your FFmpeg installation should be fine per se, but manually setting the model path seems to be required for my script to run. The default path might be this one. You could also download the model file manually and point to that.

CrypticSignal commented 4 years ago

@slhck Still doesn't work after manually specifying the path. First I put the path in quotes and got this error:

C:\Users\H\Desktop\ffmpeg-script>python -m ffmpeg_quality_metrics "trim2 [fast].mkv" trim2.mp4 -m 'C:/model-file/vmaf_v0.6.1.pkl' --enable-vmaf
Could not find model at 'C:/model-file/vmaf_v0.6.1.pkl'. Please set --model-path to a valid VMAF .pkl file.

Then I removed the quotes and got the following output:

C:\Users\H\Desktop\ffmpeg-script>python -m ffmpeg_quality_metrics "trim2 [fast].mkv" trim2.mp4 -m C:/model-file/vmaf_v0.6.1.pkl --enable-vmaf
[error] running command: ffmpeg -nostdin -y -threads 1 -i trim2.mp4 -i trim2 [fast].mkv -filter_complex [1][0]scale2ref=flags=bicubic[dist][ref];[dist][ref]libvmaf=model_path=C:/model-file/vmaf_v0.6.1.pkl:phone_model=0:log_path=C\\:/Users/H/AppData/Local/Temp/3irrkkab-vmaf.txt:log_fmt=json:psnr=1:ssim=1:ms_ssim=1 -an -f null NUL

Then there' the FFmpeg configuration info and below that I get:

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'trim2.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.20.100
  Duration: 00:00:06.99, start: 0.020998, bitrate: 2295 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 2232 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
Input #1, matroska,webm, from 'trim2 [fast].mkv':
  Metadata:
    COMPATIBLE_BRANDS: isomiso2avc1mp41
    MAJOR_BRAND     : isom
    MINOR_VERSION   : 512
    ENCODER         : Lavf58.20.100
  Duration: 00:00:05.02, start: 0.000000, bitrate: 2887 kb/s
    Stream #1:0: Video: h264 (High), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], 25 fps, 25 tbr, 1k tbn, 50 tbc (default)
    Metadata:
      HANDLER_NAME    : VideoHandler
      ENCODER         : Lavc58.35.100 libx264
      DURATION        : 00:00:05.000000000
    Stream #1:1(eng): Audio: aac (LC), 44100 Hz, stereo, fltp (default)
    Metadata:
      HANDLER_NAME    : SoundHandler
      DURATION        : 00:00:05.015000000
Stream mapping:
  Stream #0:0 (h264) -> scale2ref:ref
  Stream #1:0 (h264) -> scale2ref:default
  libvmaf -> Stream #0:0 (wrapped_avframe)
Output #0, null, to 'NUL':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.20.100
    Stream #0:0: Video: wrapped_avframe, yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc (default)
    Metadata:
      encoder         : Lavc58.35.100 wrapped_avframe
[Parsed_libvmaf_1 @ 00000213e1855b40] libvmaf encountered an error, check log for details
Error while filtering: Invalid argument
Failed to inject frame into filter network: Invalid argument
Error while processing the decoded data for stream #0:0
Conversion failed!

I even changed a section of main.py to hardcore the PATH in the if IS_WIN: block as well as on line 303 for good measure:

    if cli_args.enable_vmaf:
        if not cli_args.model_path:
            if IS_WIN:
                model_path = 'C:/model-file/vmaf_v0.6.1.pkl'
                print_stderr(
                    "Cannot not automatically determine VMAF model path under Windows."
                    "Please specify the --model-path manually."
                )
            else:
                if has_brew():
                    model_path = os.path.join(get_brewed_model_path(), "vmaf_v0.6.1.pkl")
                else:
                    print_stderr(
                        "Could not automatically determine VMAF model path, since it was not installed using Homebrew. "
                        "Please specify the --model-path manually or install ffmpeg with Homebrew."
                    )
                    sys.exit(1)
        else:
            model_path = 'C:/model-file/vmaf_v0.6.1.pkl'
slhck commented 4 years ago

Hmm, hardcoding the path is what I would have suggested next, just to figure out how the path needs to be formatted for it to be passed to VMAF correctly. See, the issue is that the slashes need to be properly escaped. I'll try to test it tomorrow.

CrypticSignal commented 4 years ago

@slhck Isn't it only backslashes that need to be escaped? I used forward slashes. Also, when specifying the path using -m, does the path need to be in quotes (-m 'C:/model-file/vmaf_v0.6.1.pkl') or without quotes (-m C:/model-file/vmaf_v0.6.1.pkl)? Without quotes, it seemed to "work" because FFmpeg tried to do something, but there's the error:

Error while filtering: Invalid argument
Failed to inject frame into filter network: Invalid argument

As you can see in my last comment.

slhck commented 4 years ago

I think it would be safer to put it in quotes, but it should not matter. What is the command that is shown when you run the command with the -v switch?

slhck commented 4 years ago

Oh, I see the path should be C\\:path/to/.pkl, as mentioned in one of my earlier comments. Can you please try that instead?

CrypticSignal commented 4 years ago

@slhck Here's the output that I get when specifying the path in the format C\\:path/to/.pkl

C:\Users\H\Desktop>ffmpeg_quality_metrics -v -ev -m "C\\:Users/H/Desktop/vmaf_v0.6.1.pkl" dist.mkv ref.mp4
Could not find model at C\\:Users/H/Desktop/vmaf_v0.6.1.pkl. Please set --model-path to a valid VMAF .pkl file.

I also tried "C:\\Users/H/Desktop/vmaf_v0.6.1.pkl" (colon after the forward slashes) instead of "C\\:Users/H/Desktop/vmaf_v0.6.1.pkl" and I get the following error:

C:\Users\H\Desktop>ffmpeg_quality_metrics -v -ev -m "C:\\Users/H/Desktop/vmaf_v0.6.1.pkl" dist.mkv ref.mp4
Writing temporary VMAF information to: C:\Users\H\AppData\Local\Temp\q1fnipri-vmaf.txt
[cmd] ffmpeg -nostdin -y -threads 1 -i ref.mp4 -i dist.mkv -filter_complex [1][0]scale2ref=flags=bicubic[dist][ref];[dist][ref]libvmaf=model_path=C:\\Users/H/Desktop/vmaf_v0.6.1.pkl:phone_model=0:log_path=C\\:/Users/H/AppData/Local/Temp/q1fnipri-vmaf.txt:log_fmt=json:psnr=1:ssim=1:ms_ssim=1 -an -f null NUL
Traceback (most recent call last):
  File "c:\users\h\appdata\local\programs\python\python38\lib\runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\h\appdata\local\programs\python\python38\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\H\AppData\Local\Programs\Python\Python38\Scripts\ffmpeg_quality_metrics.exe\__main__.py", line 7, in <module>
  File "c:\users\h\appdata\local\programs\python\python38\lib\site-packages\ffmpeg_quality_metrics\__main__.py", line 303, in main
    ret["vmaf"] = calc_vmaf(
  File "c:\users\h\appdata\local\programs\python\python38\lib\site-packages\ffmpeg_quality_metrics\__main__.py", line 137, in calc_vmaf
    raise e
  File "c:\users\h\appdata\local\programs\python\python38\lib\site-packages\ffmpeg_quality_metrics\__main__.py", line 126, in calc_vmaf
    run_command(cmd, dry_run, verbose)
  File "c:\users\h\appdata\local\programs\python\python38\lib\site-packages\ffmpeg_quality_metrics\__main__.py", line 71, in run_command
    process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
  File "c:\users\h\appdata\local\programs\python\python38\lib\subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "c:\users\h\appdata\local\programs\python\python38\lib\subprocess.py", line 1307, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified

The file is indeed in the Desktop path as you can see here in the attached image. image

In these examples, I'm using the version that I installed with pip so no hardcoding has been done in __main__.py

slhck commented 4 years ago

Ah, yeah, the backslashes should come before the colon. Please try:

C\\:/Users/H/Desktop/vmaf_v0.6.1.pkl

This includes a forward slash after the colon. It’s the same format as the temporary file name.

Sorry for only being able to provide trial and error type suggestions. I have no Windows machine available.

CrypticSignal commented 4 years ago

@slhck Same thing: 

C:\Users\H\Desktop>ffmpeg_quality_metrics -v -ev -m "C\\:/Users/H/Desktop/vmaf_v0.6.1.pkl" dist.mkv ref.mp4
Could not find model at C\\:/Users/H/Desktop/vmaf_v0.6.1.pkl. Please set --model-path to a valid VMAF .pkl file.

By the way, in the previous comment, the output when doing "C:\\Users/H/Desktop/vmaf_v0.6.1.pkl" is actually: 

C:\Users\H\Desktop>ffmpeg_quality_metrics -v -ev -m "C:\\Users/H/Desktop/vmaf_v0.6.1.pkl" dist.mkv ref.mp4
Writing temporary VMAF information to: C:\Users\H\AppData\Local\Temp\dhhutgp2-vmaf.txt
[cmd] ffmpeg -nostdin -y -threads 1 -i ref.mp4 -i dist.mkv -filter_complex [1][0]scale2ref=flags=bicubic[dist][ref];[dist][ref]libvmaf=model_path=C:\\Users/H/Desktop/vmaf_v0.6.1.pkl:phone_model=0:log_path=C\\:/Users/H/AppData/Local/Temp/dhhutgp2-vmaf.txt:log_fmt=json:psnr=1:ssim=1:ms_ssim=1 -an -f null NUL
[error] running command: ffmpeg -nostdin -y -threads 1 -i ref.mp4 -i dist.mkv -filter_complex [1][0]scale2ref=flags=bicubic[dist][ref];[dist][ref]libvmaf=model_path=C:\\Users/H/Desktop/vmaf_v0.6.1.pkl:phone_model=0:log_path=C\\:/Users/H/AppData/Local/Temp/dhhutgp2-vmaf.txt:log_fmt=json:psnr=1:ssim=1:ms_ssim=1 -an -f null NUL
ffmpeg version git-2020-07-15-a54b367 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9.3.1 (GCC) 20200621
  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libsrt --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libgsm --disable-w32threads --enable-libmfx --enable-ffnvcodec --enable-cuda-llvm --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt --enable-amf
  libavutil      56. 55.100 / 56. 55.100
  libavcodec     58. 96.100 / 58. 96.100
  libavformat    58. 48.100 / 58. 48.100
  libavdevice    58. 11.101 / 58. 11.101
  libavfilter     7. 87.100 /  7. 87.100
  libswscale      5.  8.100 /  5.  8.100
  libswresample   3.  8.100 /  3.  8.100
  libpostproc    55.  8.100 / 55.  8.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'ref.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.20.100
  Duration: 00:00:06.99, start: 0.020998, bitrate: 2295 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 2232 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
Input #1, matroska,webm, from 'dist.mkv':
  Metadata:
    COMPATIBLE_BRANDS: isomiso2avc1mp41
    MAJOR_BRAND     : isom
    MINOR_VERSION   : 512
    ENCODER         : Lavf58.20.100
  Duration: 00:00:05.02, start: 0.000000, bitrate: 5617 kb/s
    Stream #1:0: Video: h264 (Constrained Baseline), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], 25 fps, 25 tbr, 1k tbn, 50 tbc (default)
    Metadata:
      HANDLER_NAME    : VideoHandler
      ENCODER         : Lavc58.35.100 libx264
      DURATION        : 00:00:05.000000000
    Stream #1:1(eng): Audio: aac (LC), 44100 Hz, stereo, fltp (default)
    Metadata:
      HANDLER_NAME    : SoundHandler
      DURATION        : 00:00:05.015000000
Stream mapping:
  Stream #0:0 (h264) -> scale2ref:ref
  Stream #1:0 (h264) -> scale2ref:default
  libvmaf -> Stream #0:0 (wrapped_avframe)
Output #0, null, to 'NUL':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.48.100
    Stream #0:0: Video: wrapped_avframe, yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc (default)
    Metadata:
      encoder         : Lavc58.96.100 wrapped_avframe
[Parsed_libvmaf_1 @ 00000225c7718c00] libvmaf encountered an error, check log for details
Error while filtering: Invalid argument
Failed to inject frame into filter network: Invalid argument
Error while processing the decoded data for stream #0:0
Conversion failed!
slhck commented 4 years ago

I think I know what's going on. The \\ only needs to be there for ffmpeg itself. But when checking if the file exists:

if not os.path.isfile(model_path):
    print_stderr(f"Could not find model at {model_path}. Please set --model-path to a valid VMAF .pkl file.")
    sys.exit(1)

These backslashes shouldn't be there. Can you please try commenting out or removing this part, and running it again?

If that works, I simply have to adapt this check for Windows.

CrypticSignal commented 4 years ago

In what format should I specify the path (using -m) after removing that block of code?

slhck commented 4 years ago

Actually, I just realized that this issue came up before and was fixed via this commit. But this code was never applied to the model path.

I created a new branch here: https://github.com/slhck/ffmpeg-quality-metrics/tree/fix-windows-path

Does this one work if you run it with the following?

ffmpeg_quality_metrics -v -ev -m "C:/Users/H/Desktop/vmaf_v0.6.1.pkl" dist.mkv ref.mp4
CrypticSignal commented 4 years ago

@slhck The doesn't work either, both with the version where I commented out that code, where I used python -m ffmpeg_quality_metrics, and with the pip installation as you can see here:

C:\Users\H\Desktop>python -m ffmpeg_quality_metrics -v -ev -m "C:/Users/H/Desktop/vmaf_v0.6.1.pkl" dist.mkv ref.mp4
Writing temporary VMAF information to: C:\Users\H\AppData\Local\Temp\kqxkewjk-vmaf.txt
[cmd] ffmpeg -nostdin -y -threads 1 -i ref.mp4 -i dist.mkv -filter_complex [1][0]scale2ref=flags=bicubic[dist][ref];[dist][ref]libvmaf=model_path=C:/Users/H/Desktop/vmaf_v0.6.1.pkl:phone_model=0:log_path=C\\:/Users/H/AppData/Local/Temp/kqxkewjk-vmaf.txt:log_fmt=json:psnr=1:ssim=1:ms_ssim=1 -an -f null NUL
[error] running command: ffmpeg -nostdin -y -threads 1 -i ref.mp4 -i dist.mkv -filter_complex [1][0]scale2ref=flags=bicubic[dist][ref];[dist][ref]libvmaf=model_path=C:/Users/H/Desktop/vmaf_v0.6.1.pkl:phone_model=0:log_path=C\\:/Users/H/AppData/Local/Temp/kqxkewjk-vmaf.txt:log_fmt=json:psnr=1:ssim=1:ms_ssim=1 -an -f null NUL
ffmpeg version git-2020-07-15-a54b367 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9.3.1 (GCC) 20200621
  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libsrt --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libgsm --disable-w32threads --enable-libmfx --enable-ffnvcodec --enable-cuda-llvm --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt --enable-amf
  libavutil      56. 55.100 / 56. 55.100
  libavcodec     58. 96.100 / 58. 96.100
  libavformat    58. 48.100 / 58. 48.100
  libavdevice    58. 11.101 / 58. 11.101
  libavfilter     7. 87.100 /  7. 87.100
  libswscale      5.  8.100 /  5.  8.100
  libswresample   3.  8.100 /  3.  8.100
  libpostproc    55.  8.100 / 55.  8.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'ref.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.20.100
  Duration: 00:00:06.99, start: 0.020998, bitrate: 2295 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 2232 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
Input #1, matroska,webm, from 'dist.mkv':
  Metadata:
    COMPATIBLE_BRANDS: isomiso2avc1mp41
    MAJOR_BRAND     : isom
    MINOR_VERSION   : 512
    ENCODER         : Lavf58.20.100
  Duration: 00:00:05.02, start: 0.000000, bitrate: 5617 kb/s
    Stream #1:0: Video: h264 (Constrained Baseline), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], 25 fps, 25 tbr, 1k tbn, 50 tbc (default)
    Metadata:
      HANDLER_NAME    : VideoHandler
      ENCODER         : Lavc58.35.100 libx264
      DURATION        : 00:00:05.000000000
    Stream #1:1(eng): Audio: aac (LC), 44100 Hz, stereo, fltp (default)
    Metadata:
      HANDLER_NAME    : SoundHandler
      DURATION        : 00:00:05.015000000
Stream mapping:
  Stream #0:0 (h264) -> scale2ref:ref
  Stream #1:0 (h264) -> scale2ref:default
  libvmaf -> Stream #0:0 (wrapped_avframe)
Output #0, null, to 'NUL':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.48.100
    Stream #0:0: Video: wrapped_avframe, yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc (default)
    Metadata:
      encoder         : Lavc58.96.100 wrapped_avframe
[Parsed_libvmaf_1 @ 00000162ac255e40] libvmaf encountered an error, check log for details
Error while filtering: Invalid argument
Failed to inject frame into filter network: Invalid argument
Error while processing the decoded data for stream #0:0
Conversion failed!

C:\Users\H\Desktop>ffmpeg_quality_metrics -v -ev -m "C:/Users/H/Desktop/vmaf_v0.6.1.pkl" dist.mkv ref.mp4
Writing temporary VMAF information to: C:\Users\H\AppData\Local\Temp\1d_m9zfo-vmaf.txt
[cmd] ffmpeg -nostdin -y -threads 1 -i ref.mp4 -i dist.mkv -filter_complex [1][0]scale2ref=flags=bicubic[dist][ref];[dist][ref]libvmaf=model_path=C:/Users/H/Desktop/vmaf_v0.6.1.pkl:phone_model=0:log_path=C\\:/Users/H/AppData/Local/Temp/1d_m9zfo-vmaf.txt:log_fmt=json:psnr=1:ssim=1:ms_ssim=1 -an -f null NUL
[error] running command: ffmpeg -nostdin -y -threads 1 -i ref.mp4 -i dist.mkv -filter_complex [1][0]scale2ref=flags=bicubic[dist][ref];[dist][ref]libvmaf=model_path=C:/Users/H/Desktop/vmaf_v0.6.1.pkl:phone_model=0:log_path=C\\:/Users/H/AppData/Local/Temp/1d_m9zfo-vmaf.txt:log_fmt=json:psnr=1:ssim=1:ms_ssim=1 -an -f null NUL
ffmpeg version git-2020-07-15-a54b367 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9.3.1 (GCC) 20200621
  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libsrt --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libgsm --disable-w32threads --enable-libmfx --enable-ffnvcodec --enable-cuda-llvm --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt --enable-amf
  libavutil      56. 55.100 / 56. 55.100
  libavcodec     58. 96.100 / 58. 96.100
  libavformat    58. 48.100 / 58. 48.100
  libavdevice    58. 11.101 / 58. 11.101
  libavfilter     7. 87.100 /  7. 87.100
  libswscale      5.  8.100 /  5.  8.100
  libswresample   3.  8.100 /  3.  8.100
  libpostproc    55.  8.100 / 55.  8.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'ref.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.20.100
  Duration: 00:00:06.99, start: 0.020998, bitrate: 2295 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 2232 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
Input #1, matroska,webm, from 'dist.mkv':
  Metadata:
    COMPATIBLE_BRANDS: isomiso2avc1mp41
    MAJOR_BRAND     : isom
    MINOR_VERSION   : 512
    ENCODER         : Lavf58.20.100
  Duration: 00:00:05.02, start: 0.000000, bitrate: 5617 kb/s
    Stream #1:0: Video: h264 (Constrained Baseline), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], 25 fps, 25 tbr, 1k tbn, 50 tbc (default)
    Metadata:
      HANDLER_NAME    : VideoHandler
      ENCODER         : Lavc58.35.100 libx264
      DURATION        : 00:00:05.000000000
    Stream #1:1(eng): Audio: aac (LC), 44100 Hz, stereo, fltp (default)
    Metadata:
      HANDLER_NAME    : SoundHandler
      DURATION        : 00:00:05.015000000
Stream mapping:
  Stream #0:0 (h264) -> scale2ref:ref
  Stream #1:0 (h264) -> scale2ref:default
  libvmaf -> Stream #0:0 (wrapped_avframe)
Output #0, null, to 'NUL':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.48.100
    Stream #0:0: Video: wrapped_avframe, yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc (default)
    Metadata:
      encoder         : Lavc58.96.100 wrapped_avframe
[Parsed_libvmaf_1 @ 000001cf91d98bc0] libvmaf encountered an error, check log for details
Error while filtering: Invalid argument
Failed to inject frame into filter network: Invalid argument
Error while processing the decoded data for stream #0:0
Conversion failed!

Or do you want me to edit __main__.py with your commit before running that command? If you want me to edit the file, then shouldn't I be entering python -m ffmpeg_quality_metrics rather than just ffmpeg_quality_metrics?

CrypticSignal commented 4 years ago

Editing __main__.py with your commit and running it with python -m ffmpeg_quality_metrics gives:

C:\Users\H\Desktop>python -m ffmpeg_quality_metrics -v -ev -m "C:/Users/H/Desktop/vmaf_v0.6.1.pkl" dist.mkv ref.mp4
Writing temporary VMAF information to: C:\Users\H\AppData\Local\Temp\20k_n0r4-vmaf.txt
[cmd] ffmpeg -nostdin -y -threads 1 -i ref.mp4 -i dist.mkv -filter_complex [1][0]scale2ref=flags=bicubic[dist][ref];[dist][ref]libvmaf=model_path=C:/Users/H/Desktop/vmaf_v0.6.1.pkl:phone_model=0:log_path=C\\:/Users/H/AppData/Local/Temp/20k_n0r4-vmaf.txt:log_fmt=json:psnr=1:ssim=1:ms_ssim=1 -an -f null NUL
[error] running command: ffmpeg -nostdin -y -threads 1 -i ref.mp4 -i dist.mkv -filter_complex [1][0]scale2ref=flags=bicubic[dist][ref];[dist][ref]libvmaf=model_path=C:/Users/H/Desktop/vmaf_v0.6.1.pkl:phone_model=0:log_path=C\\:/Users/H/AppData/Local/Temp/20k_n0r4-vmaf.txt:log_fmt=json:psnr=1:ssim=1:ms_ssim=1 -an -f null NUL
ffmpeg version git-2020-07-15-a54b367 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9.3.1 (GCC) 20200621
  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libsrt --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libgsm --disable-w32threads --enable-libmfx --enable-ffnvcodec --enable-cuda-llvm --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt --enable-amf
  libavutil      56. 55.100 / 56. 55.100
  libavcodec     58. 96.100 / 58. 96.100
  libavformat    58. 48.100 / 58. 48.100
  libavdevice    58. 11.101 / 58. 11.101
  libavfilter     7. 87.100 /  7. 87.100
  libswscale      5.  8.100 /  5.  8.100
  libswresample   3.  8.100 /  3.  8.100
  libpostproc    55.  8.100 / 55.  8.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'ref.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.20.100
  Duration: 00:00:06.99, start: 0.020998, bitrate: 2295 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 2232 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
Input #1, matroska,webm, from 'dist.mkv':
  Metadata:
    COMPATIBLE_BRANDS: isomiso2avc1mp41
    MAJOR_BRAND     : isom
    MINOR_VERSION   : 512
    ENCODER         : Lavf58.20.100
  Duration: 00:00:05.02, start: 0.000000, bitrate: 5617 kb/s
    Stream #1:0: Video: h264 (Constrained Baseline), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], 25 fps, 25 tbr, 1k tbn, 50 tbc (default)
    Metadata:
      HANDLER_NAME    : VideoHandler
      ENCODER         : Lavc58.35.100 libx264
      DURATION        : 00:00:05.000000000
    Stream #1:1(eng): Audio: aac (LC), 44100 Hz, stereo, fltp (default)
    Metadata:
      HANDLER_NAME    : SoundHandler
      DURATION        : 00:00:05.015000000
Stream mapping:
  Stream #0:0 (h264) -> scale2ref:ref
  Stream #1:0 (h264) -> scale2ref:default
  libvmaf -> Stream #0:0 (wrapped_avframe)
Output #0, null, to 'NUL':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.48.100
    Stream #0:0: Video: wrapped_avframe, yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc (default)
    Metadata:
      encoder         : Lavc58.96.100 wrapped_avframe
[Parsed_libvmaf_1 @ 00000259a7d78440] libvmaf encountered an error, check log for details
Error while filtering: Invalid argument
Failed to inject frame into filter network: Invalid argument
Error while processing the decoded data for stream #0:0
Conversion failed!

Looking at the error, it might be an FFmpeg issue rather than a path issue?

CrypticSignal commented 4 years ago

Ugh, this is frustrating. I'd love to get this to work so I can add the VMAF value to the table created by my libx264/libx265 presets comparing program: https://github.com/BassThatHertz/compare-x264-or-x265-presets

CrypticSignal commented 4 years ago

Just tried running the following FFmpeg command by itself: ffmpeg -nostdin -y -threads 1 -i ref.mp4 -i dist.mkv -filter_complex [1][0]scale2ref=flags=bicubic[dist][ref];[dist][ref]libvmaf=model_path=C\\:/Users/H/Desktop/vmaf_v0.6.1.pkl:phone_model=0:log_path=C\\:/Users/H/AppData/Local/Temp/20k_n0r4-vmaf.txt:log_fmt=json:psnr=1:ssim=1:ms_ssim=1 -an -f null NUL and I get the error: VmafException: Error loading SVM model

slhck commented 4 years ago

Hmm, yeah this is annoying indeed. Sorry I cannot be of more help.

It's weird that you get a different error from the manual run. The error is raised here: https://github.com/Netflix/vmaf/blob/0ad6566f3fa6ccb7126877655d56a39fe9d4b841/libvmaf/src/svm.cpp#L502 — it could be that it can't load the file from there either.

@cdgriffith Do you have an idea what could cause these issues? Your Windows fix unfortunately didn't catch this part of the code.

CrypticSignal commented 4 years ago

No worries. Considering I don't get the Could not find model at<path>. Please set --model-path to a valid VMAF .pkl file. error anymore, doesn't this mean that the path is correct and rather the issue is elsewhere? Looking at the error, it seems like the issue is related to FFmpeg/libvmaf?

[Parsed_libvmaf_1 @ 00000259a7d78440] libvmaf encountered an error, check log for details
Error while filtering: Invalid argument
Failed to inject frame into filter network: Invalid argument
slhck commented 4 years ago

Yeah, the "Could not find model at" error was caused by Python. I am checking whether the file path exists, but it expects a "normal" path without escapes to be passed via command-line. So if you pass C\\:/foo.txt, it'll fail — it can only check C:/foo.txt.

The proper solution would be to have the user pass a normal path without escapes, and handle the escaping in the code itself, just like it's done for the other logfile paths.

Given that you're getting an FFmpeg/libvmaf error, the issue seems to be related to the library itself. The error is very vague though. Is there anything in the log text file?

Alternatively you may want to run the Docker version of this script.

CrypticSignal commented 4 years ago

  1. The log file should be in C:/Users/H/AppData/Local/Temp/0bss83a7-vmaf.txt, correct? I see no such file there though.

  2. For any subsequent testing, should I use the stock version of ffmpeg-quality-metrics installed with pip install ffmpeg_quality_metrics or should I use the cloned repository with python -m with the contents of __main__.py changed to match this version of the file?

slhck commented 4 years ago

  1. If you run it manually, yes. If you run it via ffmpeg-quality-metrics, it'll delete the logfile after execution ( os.remove(temp_file_name_vmaf)`).

  2. I just pushed the same change to master, and released it as 0.3.12, so you can use the stock version for the time being. Let's see if someone else has an idea.

CrypticSignal commented 4 years ago

There's no .txt file when running it manually either, and okay.

CrypticSignal commented 4 years ago

The issue was that you also need a vmaf_v0.6.1.pkl.model file in the same directory that vmaf_v0.6.1.pkl is in. All this trial-and-error when I simply needed a file.

slhck commented 4 years ago

Well that is certainly unexpected and undocumented. I'll see if I can fix that upstream.

CrypticSignal commented 4 years ago

Well, adding the file fixed it when using libvmaf with FFmpeg directly with the following command: ffmpeg -loglevel warning -stats -i dist.mp4 -i ref.mp4 -lavfi libvmaf=model_path=vmaf_v0.6.1.pkl:log_path=vmaf.json:log_fmt=json -report -f null -

However, when using ffmpeg_quality_metrics, Although I no longer get the error that I was getting before, when I enter ffmpeg_quality_metrics -v -ev -m vmaf_v0.6.1.pkl dist.mkv 0.mkv, it gets stuck with a blinking cursor thing and doesn't do anything. See the attached image.

image

So the adventure continues. Looking at Task Manager, I can see ffmpeg.exe with high CPU usage during this phase, but I've waited longer than the time the vmaf calculation takes when running the FFmpeg command manually, so something isn't right.

slhck commented 4 years ago

Can you try with a short video to see if it completes? I don't think there's much of an overhead running it with Python.

CrypticSignal commented 4 years ago

Just tried with 7 second files. Not sure how long it should take but it took longer than than one minute for sure. Two issues:

  1. You can't see the calculation progress. All you see is what you can see in the screenshot in my last comment. Surely this isn't expected behaviour, because this isn't good for user feedback - the user doesn't know what's going on.
  2. The final VMAF score isn't shown. See the attached file which shows the JSON output (I saved it as a .txt file as GitHub does not allow you to attach a JSON file). output.txt

Also, is it normal that some frames have a VMAF of 0? I don't know how the algorithm works, but some frames having a quality of 0 doesn't seem right. I would've thought that a value of 0 is impossible unless it's a completely black frame or something.

slhck commented 4 years ago

You can't see the calculation progress

That's right. Getting the progress with ffmpeg is not trivial. I implemented that here, but it'd have to ported to this project, a progress bar has to be added, etc. PRs are welcome → #12

The final VMAF score isn't shown.

True. Only per-frame metrics are computed. Since the output can be both JSON and CSV, I opted for keeping it as simple as possible, as with CSV, you cannot meaningfully show per-frame data and global aggregate data (it's not tidy).

That said, it could be added to the JSON output. Again, PRs are welcome → #13

is it normal that some frames have a VMAF of 0?

No, that should not be the case. Even if black frames are compared, that should get you a quality of 100 since they are equal.

CrypticSignal commented 4 years ago

Getting the progress with ffmpeg is not trivial.

FFmpeg shows the progress when using libvmaf. I use libvmaf in my own command line program and I have attached an example of the output where you can see that it shows the progress in terms of how far into the file it currently is.

image

True. Only per-frame metrics are computed.

libvmaf shows the final VMAF score at the end as you can see in the screenshot below. You can also try my command line program which uses FFmpeg libvmaf and see for yourself.

image

slhck commented 4 years ago

Getting the progress with ffmpeg is not trivial.

FFmpeg shows the progress when using libvmaf. I use libvmaf in my own command line program and I have attached an example of the output where you can see that it shows the progress in terms of how far into the file it currently is. image

Yes, I am aware of that, but this is just the current frame number. As you can see in the code I linked to, you have to parse that properly and show it to the user, ideally not just based on the current frame number, but also based on the overall duration of the file, to get a progress percentage and ETA. For me, just displaying the frame number wouldn't be much better in terms of UI, other than indicating that there is some kind of progress.

True. Only per-frame metrics are computed.

libvmaf shows the final VMAF score at the end. image

I should have said "only per-frame metrics are computed by this program". However, as mentioned, adding this overall score to a CSV report didn't make sense at the time of implementation, and you may want other stats displayed, that's why I proposed additional aggregate statistics in #13.

slhck commented 4 years ago

Added global statistics in v0.4.0.

CrypticSignal commented 4 years ago

Not just the frame number but you can also see the progress in terms of the duration into the file (time=00:00:05.99 in the screenshot above). But yes, not as good as seeing a percentage or ETA.

I've got a lot to learn so the following is probably an amateur approach but you can do it like the following I think?:

slhck commented 4 years ago

This is one approach, but it can be simplified by directly reading the ffmpeg stderr (this is where it prints its messages, not stdout) and parsing it from there. Please check the link from earlier — this is how I implemented it in another program. I just have to port that function to this one.

CrypticSignal commented 4 years ago

Just curious: on macOS and Linux, is a vmaf_v0.6.1.pkl.model file not needed? On Windows, I also needed a vmaf_v0.6.1.pkl.model file in the same path as the model file (vmaf_v0.6.1.pkl)

slhck commented 4 years ago

No, both files need to be there, apparently:

➜ ls -1 /usr/local/opt/libvmaf/share/model/vmaf_v0.6.1.pkl*
/usr/local/opt/libvmaf/share/model/vmaf_v0.6.1.pkl
/usr/local/opt/libvmaf/share/model/vmaf_v0.6.1.pkl.model

I just wasn't aware of it.

slhck commented 4 years ago

Another PS: I updated the README to add instructions for specifying the model path. I hope that will help other users.