Incorrect VMAF calculations while use of "-r" switch in FFmpeg command line

[jeryll7]

• as for the latest incorrect calculations of vmaf score - the problem lies in the added -r switch into FFmpeg command line : 2023-02-18 17:25:44 ffmpeg -hide_banner -nostdin -r 23.81 -i reencoded.mkv -r 23.98 -i original.mkv -frames:v 480 -lavfi [0:v]trim=start=60,settb=AVTB,setpts=PTS-STARTPTS[main];[1:v]trim=start=60,settb=AVTB,setpts=PTS-STARTPTS[ref];[main][ref]libvmaf=eof_action=endall:n_threads=23:log_fmt=json:log_path='Z\:/TMP/FFMetrics/825e1c2d-64cb-455e-be3c-75a57c515c8b.log':pool=Mean:model=version=vmaf_v0.6.1 -f null - -- you are probably extracting frame rate info via FFmpeg focusing on video stream, and I'm not sure its (FFmpeg frame rate detection from video stream) working correctly (at least for Matroska (.mkv) format) -- I've done only limited number of tests with 1.4.7b1, but a lot of my reencoded MKVs are VFRs with 23.976 fps and all of them are (also) detected by FFmpeg family based players, encoders, decoders as 23.81 fps (I noticed this while using FFmpeg, FFprobe, and MPV player) -- I'm no coder, nor expert on Matroska - but here is interesting debate about frame rate detection in ffmpeg - and If I understand that thread correctly, nothing were fixed (that was 8 years ago!) -- in my simple view of a home user, you can compute correct frame rate of media file through "frames/duration" formula or switch to Mediainfo (cli or dll - whatever suits you (so far Mediainfo always show frame rate of my medias correctly)) to get the correct fps of the analyzed media -- or make use of "-r" switch optional

[fifonik]

Could you provide output of ffmpeg command for both files: ffmpeg -hide_banner -i path\to\file.ext -frames:v 30 -filter:v idet -f null - I'd like to see what ffmpeg returned in 'fps' and 'tbr'.

Also, if you pretty sure that the -r option is causing the issue, from version 1.4.5b you can adjust how FFMetrics supplying command line parameters to FFMpeg while calculating metrics:

1. Copy FFMetrics.conf.example as FFMetrics.conf
2. Open the text file in Notepad or any other plain text editor (not Word!) and remove everything but leave this (notice that comma after the section should be removed):

   {
   "Metric": {
       "Template": "-hide_banner -nostdin -r {{main-framerate}} -i {{main-src}} -r {{ref-framerate}} -i {{ref-src}} -frames:v {{duration}} -lavfi {{lavfi}} -f null -"
   }
   }

3. Now you can remove the -r {{main-framerate}} and -r {{ref-framerate}}
4. Restart FFMetrics. Now -r option should not be supplied while calling ffmpeg.

P.S. Working with VFR is asking for troubles (I do video editing so I know what am I talking about).

[jeryll7]

ffmpeg -hide_banner -i path\to\file.ext -frames:v 30 -filter:v idet -f null -

original.mkv
...
Stream #0:0(eng): Video: wrapped_avframe, yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 23.98 fps, 23.98 tbn (default)
...
[Parsed_idet_0 @ 000002533f96b340] Repeated Fields: Neither:    32 Top:     0 Bottom:     0
[Parsed_idet_0 @ 000002533f96b340] Single frame detection: TFF:     0 BFF:     0 Progressive:     0 Undetermined:    32
[Parsed_idet_0 @ 000002533f96b340] Multi frame detection: TFF:     0 BFF:     0 Progressive:     0 Undetermined:    32

reencoded.mkv
...
Stream #0:0: Video: hevc (Main 10), yuv420p10le(tv), 1920x1080, SAR 1:1 DAR 16:9, 23.81 fps, 23.81 tbr, 1k tbn (default)
...
[Parsed_idet_0 @ 00000240523f5240] Repeated Fields: Neither:    33 Top:     0 Bottom:     0
[Parsed_idet_0 @ 00000240523f5240] Single frame detection: TFF:     2 BFF:     2 Progressive:     0 Undetermined:    29
[Parsed_idet_0 @ 00000240523f5240] Multi frame detection: TFF:     0 BFF:     5 Progressive:     0 Undetermined:    28

MediaInfo.exe Y:\original.mkv (VIDEO part)
Video
ID                                        : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4.1
Format settings                          : CABAC / 4 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 4 frames
Codec ID                                 : V_MPEG4/ISO/AVC
Duration                                 : 41 min 16 s
Bit rate                                 : 9 854 kb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 23.976 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.198
Stream size                              : 2.78 GiB (85%)
Writing library                          : x264 core 142 r2453 ea0ca51

MediaInfo.exe Y:\reencoded.mkv (VIDEO part)
Video
ID                                       : 1
Format                                   : HEVC
Format/Info                              : High Efficiency Video Coding
Format profile                           : Main 10@L5.1@Main
Codec ID                                 : V_MPEGH/ISO/HEVC
Duration                                 : 41 min 15 s
Bit rate                                 : 4 132 kb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Variable
Frame rate                               : 23.976 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 10 bits
Bits/(Pixel*Frame)                       : 0.083
Stream size                              : 1.19 GiB (73%)
Writing library                          : x265 3.5+69+0-293c84ed5:[Windows][MSVC 1928][64 bit] 10bit

[jeryll7]

Just for the reference - FFmpeg command line and the result bad vmaf ffmpeg -hide_banner -nostdin -r 23.81 -i Y:/reencoded.mkv -r 23.98 -i Y:/original.mkv -frames:v 719 -lavfi [0:v]trim=start=60,settb=AVTB,setpts=PTS-STARTPTS[main];[1:v]trim=start=60,settb=AVTB,setpts=PTS-STARTPTS[ref];[main][ref]libvmaf=eof_action=endall:n_threads=23:log_fmt=json:log_path='Z\:/TMP/FFMetrics/9c274ddd-763f-4efc-83d0-0e45c3aaabe6.log':pool=Mean:model=version=vmaf_v0.6.1 -f null - result [Parsed_libvmaf_6 @ 0000029f46a7fb80] VMAF score: 6.860010

good vmaf ffmpeg -hide_banner -nostdin -r 23.98 -i Y:/reencoded.mkv -r 23.98 -i Y:/original.mkv -frames:v 719 -lavfi [0:v]trim=start=60,settb=AVTB,setpts=PTS-STARTPTS[main];[1:v]trim=start=60,settb=AVTB,setpts=PTS-STARTPTS[ref];[main][ref]libvmaf=eof_action=endall:n_threads=23:log_fmt=json:log_path='Z\:/TMP/FFMetrics/9c274ddd-763f-4efc-83d0-0e45c3aaabe6.log':pool=Mean:model=version=vmaf_v0.6.1 -f null - result [Parsed_libvmaf_6 @ 00000220abb79380] VMAF score: 96.074574

[jeryll7]

I also tried to edit FFMetrics config file as you suggested, but 1.4.7b version only throws exception on start and then quits. It only starts with the original config file from the archive... Tried different file names as well (FFMetrics.conf, FFMetrics.exe.conf, FFMetrics.config), no dice...

[fifonik]

Thanks for the info.

As a temporary solution I've just published 1.3.2 beta that is 1.3.1 release with back ported fix for the high GPU utilisation issue.

[fifonik]

What do you mean under "original config file from the archive"? Are you talking about FFMetrics.exe.config? You shoult keep this file as-is and do not touch it.

I mentioned FFMetrics.conf.example that is also in the archive. This file is an example (as extension suggests). You need to rename it to FFMetrics.conf (just remove the last .example bit from full file name), but not to FFMetrics.config, FFMetrics.exe.config, etc. The .conf file format is changing from time to time. As a result, if you use FFMetrics 1.4.7 you need to start your .conf based on .conf.example from the archive with 1.4.7.

I'm trying to keep the content of this FFMetrics.conf.example exactly as the program's internal defaults. If program not starting after you renamed the FFMetrics.conf.example to FFMetrics.conf -- please delete FFMetrics.log, start program with command line option --log-level=debug and supply the log with your .conf.

[fifonik]

Based on the ffmpeg media info, I do not think I will be able to help (other than provide an ability to modify ffmpeg arguments that is already implemented).

I'm trying to keed the program as simple as possible and not add other dependances so using mediainfo or even ffprobe is not an option. Sorry.

[jeryll7]

What do you mean under "original config file from the archive"? Are you talking about FFMetrics.exe.config? You shoult keep this file as-is and do not touch it.

I mentioned FFMetrics.conf.example that is also in the archive. This file is an example (as extension suggests). You need to rename it to FFMetrics.conf (just remove the last .example bit from full file name), but not to .config, .exe.config etc. As the .conf file format is changing from time to time. As a result, if you use FFMetrics 1.4.7 you need to start your .conf based on .conf.example from the archive with 1.4.7.

I'm trying to keep the content of this FFMetrics.conf.example exactly as internal defaults. If you renamed the FFMetrics.conf.example to FFMetrics.conf and program cannot start any longer -- please delete FFMetrics.log, start program with command line option --log-level=debug and supply the log with your .conf.

Oh, I see... OK ... done ... VMAF calculation works with the 1.4.7b just fine now. (I also replaced AVAST antivirus with BitDefender and no more exceptions)

Thank you for your help, and will close this as solved.