Incorrect metrics when Ref or Dist are Y4M

[Marzhin971]

The title says everything.

Besides that I was able to check if this was a sync or source problem with ffmpeg with those command lines :

For the ref video : ffmpeg -fflags +genpts -i .\m101.y4m -c:v copy -an -sn ref.mkv

For the dist video : ffmpeg -fflags +genpts -i m101.libaom-av1.2p.06.mkv -c:v copy -an -sn dist.mkv

FYI : I'm doing it like that because including other streams, like audio, in the mkv can lead to sync problems. So I'm generating temporary mkv files.

and launching ffmpeg/vmaf with this command : ffmpeg -i dist.mkv -i ref.mkv -lavfi "[0:v]scale=1920:1080:flags=bicubic[dist];[1:v]scale=1920:1080[ref];[dist][ref]libvmaf=model_path='f\:\vidtest\vmaf_v0.6.1.json':log_fmt=csv:n_threads=24:log_path=m101.libaom-av1.2p.06.mkv.csv:pool=mean" -f null -

The result with this command is an average VMAF of 83.246935

now if I use the same command but, with the y4m file as ref which gives : ffmpeg -i dist.mkv -i m101.y4m -lavfi "[0:v]scale=1920:1080:flags=bicubic[dist];[1:v]scale=1920:1080[ref];[dist][ref]libvmaf=model_path='f\:\vidtest\vmaf_v0.6.1.json':log_fmt=csv:n_threads=24:log_path=m101.libaom-av1.2p.06.mkv.csv:pool=mean" -f null -

the result is 46.058474, which is obviously wrong.

Now if I go fix the framerate with ffmpeg, in the vmaf calculation command, which gives this command line : ffmpeg -r 24000/1001 -i dist.mkv -r 24000/1001 -i m101.y4m -lavfi "[0:v]scale=1920:1080:flags=bicubic[dist];[1:v]scale=1920:1080[ref];[dist][ref]libvmaf=model_path='f\:\vidtest\vmaf_v0.6.1.json':log_fmt=csv:n_threads=24:log_path=m101.libaom-av1.2p.06.mkv.csv:pool=mean" -f null -

The result is then back to a correct value of 83.246935

It's obvious that something is wrong with ffmpeg framerate/frame pts calculations, and we have to tell him which is the real framerate to get a correct calculation. My solution is only a workaround, or even less, let's say it's the beginning of an idea to fix that, but at least ffMetrics will hopefully have the possibility to get Y4M as input files and give correct values :)

[fifonik]

Option -r is recommended by VMAF developers and was recently added into FFMetrics. So now the option is supplied to FFMpeg by default (check this issue for details). User still can adjust this behaviour by manually editing FFMetrics config so the -r will not be used.

Unfortunately, I know nothing about the "y2m" video and have no idea if FFMpeg is able to get the fps/tbn for such videos. If FFMpeg cannot get the correct values, FFMetrics cannot supply these values back to FFMpeg during VMAF calculation and so the calculation fails and I'm unable do anything about this.

It would be possible to supply such information to FFMpeg if FFMetrics have UI where user can specify params such as frame resolution and frame rate per every file. Unfortunately, I do not have plans to implement such UI as this will ruin the main purpose of FFMetrics: "provide a simple and easy to use UI for video quality metrics calculation using FFMpeg".

The commands you provided are definitely not from FFMetrics. So it is possible that FFMpeg is reporting back framerate for y2m files and so FFMetrics is already calculating VMAF metric correctly. You can just try and report back with FFMpeg commands issued by FFMetrics.

As of now I do not know how can I help you. It would be nice if you provide some info (but report or feature request) that is related to FFMetrics. Thanks.

[Marzhin971]

Hello

Y4M incorporate all necessary specs for the video you want to get info on. You can just use : ffprobe file.y4m, if ffprobe knows, then ffmpeg will know. tbh , you'd better use ffprobe than ffmpeg to parse those informations (mediainfo as well is a great tool for that, as it will also give x264/x265 encoding parameters in a specific string)

Here's an output example from ffprobe :

Input #0, yuv4mpegpipe, from 'm101.y4m': Duration: 00:00:30.03, start: 0.000000, bitrate: 596601 kb/s Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p(progressive), 1920x1080, SAR 1:1 DAR 16:9, 23.98 fps, 23.98 tbr, 23.98 tbn

and the informations given my mediainfo for the same file :

**General Complete name : m101.y4m Format : YUV4MPEG2 File size : 2.09 GiB

Video Format : YUV Width : 1 920 pixels Height : 1 080 pixels Display aspect ratio : 16:9 Frame rate : 23.976 (24000/1001) FPS Color space : YUV Scan type : Progressive Compression mode : Lossless**

While ffprobe gives you quite sufficient informations, mediainfo is much more precise and its output can be formatted to several formats, like json or xml. in mediainfo ouput you notably get the fps numerator and denominator which are very important for the -r VMAF commandline argument.

About your question, I don't know how to provide "bug report" or "feature request" on github, I'm new to it, but you can consider it as a bug report since Y4M is a standard video file and is supposed to be correctly analyzed (sorry if my writing looks rude, english isnt my mother language). My post can be summarized by this simple phrase : "ffmetric doesnt calculate metrics correctly for Y4M files, here's an idea to fix it"

[fifonik]

sorry if my writing looks rude, english isnt my mother language

All good, no any offense taken.

I still do not understand: have you tried to use the latest version of FFMetrics with you footage? If 'yes' and you got wrong VMAF value, could you follow troubleshooting steps please? Then provide log and possible files (if they are not too big).

Thanks.

[Marzhin971]

I downloaded the latest "master" version which is 1.3.1.

Results are perfectly the same as my own tool on version 1.4.7b. I'll close this thread and sorry for the lost time