Check if vmaf works correctly with ffmpeg

[flip111]

Could there be added a little bit of code that just checks if vmaf works as it should to see if everything is setup correctly? Just as a sanity check before starting to use ffmpeg with libvmwaf.

[slhck]

This is not something that VMAF itself can do. Or what specifically do you have in mind?

In order to check if vmaf works with ffmpeg, you need to:

1. Compile ffmpeg with --enable-libvmaf
2. Run ffmpeg with ffmpeg -f lavfi -i nullsrc -f lavfi -i nullsrc -filter_complex libvmaf -frames:v 1 -f null /dev/null

Both of these need to run without errors.

One problem is that ffmpeg might not always exit with a non-zero return code if there was an issue loading VMAF. For example, if the model file cannot be found, it'll just say “Caught VmafException: Error loading model (.pkl)” and exit with 0.

[flip111]

2\. Run ffmpeg with `ffmpeg -f lavfi -i nullsrc -f lavfi -i nullsrc -filter_complex libvmaf -frames:v 1 -f null /dev/null`

Ah thanks this is already a very helpful command.

Or what specifically do you have in mind?

Just to report vmaf version, model loading went right, is able to analyze the videos. Maybe a 10 second benchmark as well. This kind of thing. But just checking with that ffmpeg and looking for the error code is already great !

EDIT: i tried it

> c:\media-autobuild_suite\local64\bin-video\ffmpeg.exe -f lavfi -i nullsrc -f lavfi -i nullsrc -filter_complex libvmaf -frames:v 1 -f null /dev/null
ffmpeg version N-96112-gb38c4959d8-g231ffd7e63+2 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 9.2.0 (Rev2, Built by MSYS2 project)
  configuration:  --disable-autodetect --enable-amf --enable-bzlib --enable-cuda --enable-cuvid --enable-d3d11va --enable-dxva2 --enable-iconv --enable-lzma --enable-nvenc --enable-zlib --enable-sdl2 --enable-ffnvcodec --enable-nvdec --enable-cuda-llvm --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libdav1d --disable-debug --enable-fontconfig --enable-libass --enable-libbluray --enable-libfreetype --enable-libmfx --enable-libmysofa --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libwavpack --enable-libwebp --enable-libxml2 --enable-libzimg --enable-libshine --enable-gpl --enable-avisynth --enable-libxvid --enable-libaom --enable-libopenmpt --enable-version3 --enable-chromaprint --enable-decklink --enable-frei0r --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libfdk-aac --enable-libflite --enable-libfribidi --enable-libgme --enable-libgsm --enable-libilbc --enable-libsvthevc --enable-libkvazaar --enable-libmodplug --enable-librtmp --enable-librubberband --enable-libtesseract --enable-libxavs --enable-libzmq --enable-libzvbi --enable-openal --enable-libvmaf --enable-libcodec2 --enable-libsrt --enable-ladspa --enable-opencl --enable-opengl --enable-libopenh264 --enable-openssl --extra-cflags=-fopenmp --extra-libs=-lgomp --extra-cflags=-DLIBTWOLAME_STATIC --extra-libs=-lstdc++ --extra-cflags=-DCACA_STATIC --extra-cflags=-DMODPLUG_STATIC --extra-cflags=-DCHROMAPRINT_NODLL --extra-libs=-lstdc++ --extra-cflags=-DZMQ_STATIC --extra-libs=-lpsapi --extra-cflags=-DLIBXML_STATIC --extra-libs=-liconv --disable-w32threads --extra-cflags=-DKVZ_STATIC_LIB --enable-nonfree
  libavutil      56. 36.101 / 56. 36.101
  libavcodec     58. 65.100 / 58. 65.100
  libavformat    58. 35.101 / 58. 35.101
  libavdevice    58.  9.101 / 58.  9.101
  libavfilter     7. 69.101 /  7. 69.101
  libswscale      5.  6.100 /  5.  6.100
  libswresample   3.  6.100 /  3.  6.100
  libpostproc    55.  6.100 / 55.  6.100
Input #0, lavfi, from 'nullsrc':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 320x240 [SAR 1:1 DAR 4:3], 25 tbr, 25 tbn, 25 tbc
Input #1, lavfi, from 'nullsrc':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #1:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 320x240 [SAR 1:1 DAR 4:3], 25 tbr, 25 tbn, 25 tbc
Stream mapping:
  Stream #0:0 (rawvideo) -> libvmaf:main
  Stream #1:0 (rawvideo) -> libvmaf:reference
  libvmaf -> Stream #0:0 (wrapped_avframe)
Press [q] to stop, [?] for help
Output #0, null, to '/dev/null':
Input model at /usr/local/share/model/vmaf_v0.6.1.pkl cannot be read successfully.
C  Metadata:
    encoder         : aLavf58.35.10ug1
h    Stream #0:0t Vm: Video: wrapped_avframe, yuv420p, 320x240 [SAR 1:1 DAR 4:3], q=2-31, 200 kb/s, af25 fps, Exc25 tbn, eptio25 tbcn (default)
    Metadata:
:       encoder         : ELavc58.65.100 wrapped_avframerro
r loadframe=    1 fps=0.8 q=-0.0 Lsize=N/A time=00:00:00.04 bitrate=N/A speed=0.0303x
ivideo:1kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead:ng  unknownmodel
 (.pkl): Trouble reading the file:/usr/local/share/model/vmaf_v0.6.1.pkl

> echo Exit Code is %errorlevel%
Exit Code is 0

Maybe it could just say here "vmaf status: error loading model file at", and then when another error occurs "vmaf status: another error", and in case of success "vmaf status: OK". This helps visually and also is easier to parse to have such a definite conclusion.

>c:\media-autobuild_suite\local64\bin-video\ffmpeg.exe -f lavfi -i nullsrc -f lavfi -i nullsrc -filter_complex libvmaf="model_path=C\\:/media-autobuild_suite/local64/share/model/vmaf_v0.6.1.pkl" -frames:v 1 -f null /dev/null
ffmpeg version N-96112-gb38c4959d8-g231ffd7e63+2 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 9.2.0 (Rev2, Built by MSYS2 project)
  configuration:  --disable-autodetect --enable-amf --enable-bzlib --enable-cuda --enable-cuvid --enable-d3d11va --enable-dxva2 --enable-iconv --enable-lzma --enable-nvenc --enable-zlib --enable-sdl2 --enable-ffnvcodec --enable-nvdec --enable-cuda-llvm --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libdav1d --disable-debug --enable-fontconfig --enable-libass --enable-libbluray --enable-libfreetype --enable-libmfx --enable-libmysofa --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libwavpack --enable-libwebp --enable-libxml2 --enable-libzimg --enable-libshine --enable-gpl --enable-avisynth --enable-libxvid --enable-libaom --enable-libopenmpt --enable-version3 --enable-chromaprint --enable-decklink --enable-frei0r --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libfdk-aac --enable-libflite --enable-libfribidi --enable-libgme --enable-libgsm --enable-libilbc --enable-libsvthevc --enable-libkvazaar --enable-libmodplug --enable-librtmp --enable-librubberband --enable-libtesseract --enable-libxavs --enable-libzmq --enable-libzvbi --enable-openal --enable-libvmaf --enable-libcodec2 --enable-libsrt --enable-ladspa --enable-opencl --enable-opengl --enable-libopenh264 --enable-openssl --extra-cflags=-fopenmp --extra-libs=-lgomp --extra-cflags=-DLIBTWOLAME_STATIC --extra-libs=-lstdc++ --extra-cflags=-DCACA_STATIC --extra-cflags=-DMODPLUG_STATIC --extra-cflags=-DCHROMAPRINT_NODLL --extra-libs=-lstdc++ --extra-cflags=-DZMQ_STATIC --extra-libs=-lpsapi --extra-cflags=-DLIBXML_STATIC --extra-libs=-liconv --disable-w32threads --extra-cflags=-DKVZ_STATIC_LIB --enable-nonfree
  libavutil      56. 36.101 / 56. 36.101
  libavcodec     58. 65.100 / 58. 65.100
  libavformat    58. 35.101 / 58. 35.101
  libavdevice    58.  9.101 / 58.  9.101
  libavfilter     7. 69.101 /  7. 69.101
  libswscale      5.  6.100 /  5.  6.100
  libswresample   3.  6.100 /  3.  6.100
  libpostproc    55.  6.100 / 55.  6.100
Input #0, lavfi, from 'nullsrc':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 320x240 [SAR 1:1 DAR 4:3], 25 tbr, 25 tbn, 25 tbc
Input #1, lavfi, from 'nullsrc':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #1:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 320x240 [SAR 1:1 DAR 4:3], 25 tbr, 25 tbn, 25 tbc
Stream mapping:
  Stream #0:0 (rawvideo) -> libvmaf:main
  Stream #1:0 (rawvideo) -> libvmaf:reference
  libvmaf -> Stream #0:0 (wrapped_avframe)
Press [q] to stop, [?] for help
Output #0, null, to '/dev/null':
  Metadata:
    encoder         : Lavf58.35.101
    Stream #0:0: Video: wrapped_avframe, yuv420p, 320x240 [SAR 1:1 DAR 4:3], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc (default)
    Metadata:
      encoder         : Lavc58.65.100 wrapped_avframe
frame=    1 fps=0.0 q=-0.0 Lsize=N/A time=00:00:00.04 bitrate=N/A speed=0.0484x
video:1kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
[libvmaf @ 00000255618ed0c0] VMAF score: 97.428043

>echo Exit Code is %errorlevel%
Exit Code is 0

Is this a success here because i got "VMAF score" or because "VMAF score: 97.428043" ? Seems a bit strange because looking at the input command source and destination are the same, so why not "VMAF score: 100"

[slhck]

This sounds like a feature request (or actually, a bug report) for the library, i.e. throwing an error that makes ffmpeg fail if the model could not be loaded.

As for VMAF not giving a score of 100 on the same signal, see: https://github.com/Netflix/vmaf/issues/76