[Bug] VMAF not available after upgrade

[fu-hsi]

After upgrade from 0.9.4b to 0.9.8b VMAF support was disabled (information from status bar and VMAF column can't be selected). I don't changed anything, I have both versions for test and the old one worked. ffmpeg-2020-11-04.

[fifonik]

Could you please:

• Close FFMetrics and delete FFMetrics.log;
• Run the program with option -log-level=debug;
• In program menu activate “Option | Write FFMpeg commands in log”;
• Add reference file;
• Add one distorted file and make it active;
• Click “Start” button;
• Exit the program
• Provide FFMetrics.log

Thanks.

[fu-hsi]

OK. It's only fragment but I think the most important:

, stderr: ffmpeg version 2020-11-04-git-cfdddec0c8-essentials_build-www.gyan.dev Copyright (c) 2000-2020 the FFmpeg developers built with gcc 10.2.0 (Rev3, Built by MSYS2 project) configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-lzma --enable-zlib --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-sdl2 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-libaom --enable-libopenjpeg --enable-libvpx --enable-libass --enable-libfreetype --enable-libfribidi --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-ffnvcodec --enable-nvdec --enable-nvenc --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-libgme --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libtheora --enable-libvo-amrwbenc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-librubberband libavutil 56. 60.100 / 56. 60.100 libavcodec 58.112.101 / 58.112.101 libavformat 58. 64.100 / 58. 64.100 libavdevice 58. 11.102 / 58. 11.102 libavfilter 7. 88.102 / 7. 88.102 libswscale 5. 8.100 / 5. 8.100 libswresample 3. 8.100 / 3. 8.100 libpostproc 55. 8.100 / 55. 8.100 (0) 2021-07-10 12:54:51 DEBUG: FF.VersionInfoParse.Started 2021-07-10 12:54:52 DEBUG: FF.VersionInfoParse.Finished: OK 2021-07-10 12:54:52 DEBUG: FF.VersionInfoGet.Finished: OK 2021-07-10 12:54:52 DEBUG: InitializeMetricsConfig started 2021-07-10 12:54:52 DEBUG: InitializeMetricsConfig finished 2021-07-10 12:54:52 DEBUG: FF.IsVMAFModelSupported.Started 2021-07-10 12:54:52 DEBUG: FF.ExecStatus.Started, -hide_banner -f lavfi -i color=c=black:s=640x480:d=1 -f lavfi -i color=c=black:s=640x480:d=1 -lavfi libvmaf='C\:/Users/[cenzored]/Downloads/FFMetrics.0.9.8b/FFMetrics/vmaf-models/vmaf_4k_v0.6.1.json' -f null - 2021-07-10 12:54:52 DEBUG: FF.ExecStatus.StdOut: Input model at C:/Users/[cenzored]/Downloads/FFMetrics.0.9.8b/FFMetrics/vmaf-models/vmaf_4k_v0.6.1.json cannot be read successfully. Caught VmafException: Error loading model (.pkl): Unknown token

No more exceptions caught.

[fifonik]

• Please provide the next FFMpeg call from the log, it is the most important one in this case.
• Where is your old version located? Is it also in your download folder?
• Does the [censored] part of path contains any non-English characters like in this issue?
• Does your ffmpeg support "json" or "pkl" models? If "pkl" -- do you have in "vmaf-models" folder .pkl + .pkl.model files?

[fifonik]

BTW, I see something strange in the log you provided: libvmaf='C:/Users/...

it must be backslash before colon: libvmaf='C\:/Users/...

Are you sure that you copied it correctly?

[fu-hsi]

Where is your old version located? Is it also in your download folder?

[censored] is just an ASCII text.

Does your ffmpeg support "json" or "pkl" models? If "pkl" -- do you have in "vmaf-models" folder .pkl + .pkl.model files?

Both type of files are available in vmaf-models. I downloaded all files from your github.

In previous version there is no such files with .json extension. I copied vmaf-models from old version to new version and now VMAF is working. I suppose your current vmaf-models from github is broken or json files break something.

[fifonik]

Hm. I just downloaded the archive from github, unpacked it Model files are in the folder and everything works with ffmpeg supported json models and ffmpeg supported pkl models. I will try to find out what is going wrong, but as I'm unable to re-produce the issue here, the chances to do this without log are slim.

[fu-hsi]

Then maybe my version doesn't support json files. But problem solved :)

[fifonik]

New version checking if ffmpeg supports json and fails. This is expected for ffmpeg without json models support.

Then it should check if the ffmpeg supports pkl models. Here something goes wrong in your case.

[fu-hsi]

I am using this version of ffmpeg because I have problems with a new one (with videostab module) and I don't wanna break something. I am happy with this version. If something works, don't touch ;)

[fifonik]

Yes, I perfectly understand that and not forcing you to update FFMpeg. I only with I have your log, as I believe some clues that can help me to fix the issue are there. Thanks.

[fu-hsi]

Sorry, I don't understand what is going on. I restored the previous folder with models. VMAF is available... I also unpacked all from zip again, also working... I don't understand this. If problem occurs again I will attach the entire log file 🤔.

[fu-hsi]

It's random. Once I launch program VMAF is not available. After restart is available... ;)

[fifonik]

What is the type of ffmpeg build you use -- static (3 exe files) or shared (3 exe files + a few dll files)? (Static is slower with ffmetrics).

You could try to download static ffmpeg build and put ffmpeg.exe aside of ffmetrics.exe. It will not affect any other scripts in your system.

Also, you can try to move the ffmetrics folder to C:\Temp and re-run the program a few times with the -log-level=debug option and no ref/distorted files added until the issue happening. Then send the log (it should not contain any paths in it).

[fu-hsi]

The problem occurs again. Log below: FFMetrics.log

[fu-hsi]

You also set constant framerate to 30fps (-frames:v 30). I use source with 50fps. Doesn't that misrepresent the results? Shouldn't we be using the source fps?

[fu-hsi]

After cleanup drive C: (Windows tool) worked again... Totally random ;)

[fifonik]

Thanks for the log. I downloaded it, feel free to delete it if you like.

I do not really understand why it does not work in your case. Based on log, everything executed as it should:

2021-07-12 09:54:16 DEBUG: FF.ExecStatus.Started, -hide_banner -f lavfi -i color=c=black:s=640x480:d=1 -f lavfi -i color=c=black:s=640x480:d=1 -lavfi "libvmaf='f\:/Video Metrics/FFMetrics/vmaf-models/vmaf_4k_v0.6.1.json'" -f null - 2021-07-12 09:54:16 DEBUG: FF.ExecStatus.StdOut: Input model at f:/Video Metrics/FFMetrics/vmaf-models/vmaf_4k_v0.6.1.json cannot be read successfully. Caught VmafException: Error loading model (.pkl): Unknown token ... 2021-07-12 09:54:16 DEBUG: FF.ExecStatus.Started, -hide_banner -f lavfi -i color=c=black:s=640x480:d=1 -f lavfi -i color=c=black:s=640x480:d=1 -lavfi "libvmaf='f\:/Video Metrics/FFMetrics/vmaf-models/vmaf_4k_v0.6.1.pkl'" -f null - 2021-07-12 09:54:18 DEBUG: FF.ExecStatus.Finished. stdout: , stderr: Timed out(-2)

I highlighted the most important lines. FFMetrics checks if json model supported with this call:

ffmpeg -hide_banner -f lavfi -i color=c=black:s=640x480:d=1 -f lavfi -i color=c=black:s=640x480:d=1 -lavfi "libvmaf='f\:/Video Metrics/FFMetrics/vmaf-models/vmaf_4k_v0.6.1.json'" -f null -

Above command fails (as your ffmpeg does not support json models) and then FFMetrics checks if pkl model supported with this call:

ffmpeg -hide_banner -f lavfi -i color=c=black:s=640x480:d=1 -f lavfi -i color=c=black:s=640x480:d=1 -lavfi "libvmaf='f\:/Video Metrics/FFMetrics/vmaf-models/vmaf_4k_v0.6.1.pkl'" -f null -

Notice that the difference only in path to model file. You can try to run the last command from command prompt and I think it will work with no issues. To be honest, I can only think of about adding a small timeout after failing call. Not sure it this helps, but I will try to create test build for you later today.

You also set constant framerate to 30fps (-frames:v 30).

This option is only used whilt getting media info and it is not changing frame rate. It is limiting input stream to 30 frames. You can turn "Options | Write FFMpeg commands into log" and full ffmpeg commands will be logged so you can copy them from log and execute directly in command prompt.

[fifonik]

I asked you already if you use static or shared ffmpeg build. Should I expect your answer? :)

If you are using static, I'd recommend to download shared build and unpack it directly into the FFMetrics folder. If this helps, then additional delay in FFMetrics should help as well.

[fu-hsi]

F:\Video Metrics>ffmpeg -hide_banner -f lavfi -i color=c=black:s=640x480:d=1 -f lavfi -i color=c=black:s=640x480:d=1 -lavfi "libvmaf='f\:/Video Metrics/FFMetrics/vmaf-models/vmaf_4k_v0.6.1.pkl'" -f null -
Input #0, lavfi, from 'color=c=black:s=640x480:d=1':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 640x480 [SAR 1:1 DAR 4:3], 25 tbr, 25 tbn, 25 tbc
Input #1, lavfi, from 'color=c=black:s=640x480:d=1':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #1:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 640x480 [SAR 1:1 DAR 4:3], 25 tbr, 25 tbn, 25 tbc
Stream mapping:
  Stream #0:0 (rawvideo) -> libvmaf:main
  Stream #1:0 (rawvideo) -> libvmaf:reference
  libvmaf -> Stream #0:0 (wrapped_avframe)
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf58.64.100
    Stream #0:0: Video: wrapped_avframe, yuv420p(progressive), 640x480 [SAR 1:1 DAR 4:3], q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc (default)
    Metadata:
      encoder         : Lavc58.112.101 wrapped_avframe
Input model at f:/Video Metrics/FFMetrics/vmaf-models/vmaf_4k_v0.6.1.pkl cannot be read successfully.
Caught VmafException: Error loading model (.pkl): Trouble reading the file:f:/Video Metrics/FFMetrics/vmaf-models/vmaf_4k_v0.6.1.pkl
[Parsed_libvmaf_0 @ 0000014bfafa0540] libvmaf encountered an error, check log for details
Error while filtering: Invalid argument
Failed to inject frame into filter network: Invalid argument
Error while processing the decoded data for stream #1:0
Conversion failed!

[fu-hsi]

I asked you already if you use static or shared ffmpeg build. Should I expect your answer? :)

I might have missed it. I use this version:

ffmpeg version 2020-11-04-git-cfdddec0c8-essentials_build-www.gyan.dev Copyright (c) 2000-2020 the FFmpeg developers built with gcc 10.2.0 (Rev3, Built by MSYS2 project)

from: https://www.gyan.dev/ffmpeg/builds/.

Only binary files was in package (.exe):

ffmpeg.exe ffplay.exe ffprobe.exe

But now your app working in background (calculates VMAF), maybe that's why the command doesn't work now.

[fifonik]

I do not know if you can execute VMAF calculation with ffmpeg a multiple times at once. Never checked this, the model file could be locked so the issues.

As for your ffmpeg build. It is quite old and not available on gyan's web site any longer, so I cannot check it out.

[fu-hsi]

Downloaded.

[fu-hsi]

The same problem. Just launched and VMAF is not available, but in the same time I use previous (newest from github) version with VMAF ;) If I am the only one with this issue, ignore it. I can live with it because somehow I manage to get it to work from time to time.

[matteocontrini]

FWIW I had a similar issue in the past, VMAF support wasn't detected although the ffmpeg build included it. I'm unable to reproduce the issue now, so I'm afraid I can't help.

[fifonik]

Are you starting one version while another is calculating vmaf? As I said, you should not do this until you check if this works by executing ffmpeg directly.

P.S. If this is the exact situation you have, then I should be able to implement a workaround in case more than one model available.

[fu-hsi]

Are you starting one version while another is calculating vmaf?

Yes, I am multitask man :) I stopped app and closed.

Launched the new one. VMAF is not available. Launched previously closed app - WMAF is not available too...

Launched disk cleaning on C: and app - WMAF is available.

Maybe you store some settings on temp folder, share settings between apps from different versions, storing last wrong settings - I don't know.

I can't help more. Now it works, so I will not turn off and finish my work. Thank you for your time. I'm sorry I took so much time.

[fifonik]

I tried to run multiple programs with different ffmpeg versions:

1. Created a separate folder and unpacked FFMetrics archive into it
2. Copied FFMetrics.exe as FFMetrics2.exe
3. Run FFMetrics.exe, added ref, distorted, unselected PSNR & SSIM, selected VMAF, selected Full Stream, saved as a project
4. Closed FFMetrics.exe
5. Put FFMpeg.exe in the folder
6. Run FFMetrics.exe (project loaded automatically)
7. "Start" VMAF calculation
8. While VMAF is calculating, Run FFMetrics2.exe, checked VMAF checkbox, closed FFMetrics2.exe; Repeated this step ~10 times

Repeated 4-8 for other FFMpeg versions

Results: ffmpeg 4.3.1 release (gyan) static pkl -- FAIL mostly ffmpeg 4.3.1 release (gyan) shared pkl -- FAIL sometimes ffmpeg 4.4 release (gyan) static json -- OK (never seen an error) ffmpeg 4.4 release (gyan) shared json -- OK (never seen an error)

Unfortunately, I do not think I can do anything about this.

[fifonik]

I can try to increase timeout for such calls in the next version.

P.S. Already increased timeouts for getting version info process and getting supported VMAF model types from 2s to 5s

[fifonik]

Timeout increased in 0.9.9b. Could you check if this helped with your issue please?

[fu-hsi]

Of Course. 0.9.8b - It's not working, as expected ;) 0.9.9b - Launched 3 times, works and VMAF detects well :)

I did not check other new features.

[fifonik]

Thanks for checking it out. Please take you time to "play" with the new build and report back if you find other issues.

[fu-hsi]

First bug #64. After choosing reference file, "Add files" is still disabled (drag & drop works). In previous version this button was always enabled.