Different VMAF result with the latest versions of ab-av1

[Werve]

I upgraded from an old version (I think it was the 0.7.12) to 0.8.0 and noticed strange results in my usual workflow so I went deeper and ran a test using the same command and file again: ab-av1 vmaf --reference “pathfile1” --distorted “pathfile2”

Using ab-av1 0.8.0 the result is: 39.40909

Using ab-av1 0.8.0 and --vmaf ts_sync_mode=default the result is: 41.911076

Using ab-av1 0.7.19 the result is: 39.40909

Using ab-av1 0.7.12 the result is: 97.80562

Using ab-av1 0.7.15 the result is: 97.80562

Using ab-av1 0.7.17 the result is: 39.40909

Using FFMetrics 1.5b13 (that detect ffmpeg 6.1.1 in path): Mean: 97.805618

Using MSU VQMT Free 14.1 (with the 0.6.1 vmaf model setting and cubic upscale to 1920x1080 just in case): Mean: 97.436417

Since two external tools report a much higher vmaf I think there is an error with the latest versions of ab-av1, or is it an intended result?

Also i noticed that the 2 video files used have different duration, so i will manually compare frames to figure out better (not ab-av1 fault).

[alexheretic]

This is most likely related to #215 which was released with v0.7.17.

You could try running with debug logging to see the ffmpeg call and reproduce calling ffmpeg directly. Then modify the vmaf args to see what impacts the analysis here.

[Werve]

I enabled debug logging and saw ab-av1 0.8.0 makes a call like this: ffmpeg -i filepath2 -i filepath1 -filter_complex [0:v]format=yuv420p10le,setpts=PTS-STARTPTS,settb=AVTB[dis];[1:v]format=yuv420p10le,setpts=PTS-STARTPTS,settb=AVTB[ref];[dis][ref]libvmaf=shortest=true:ts_sync_mode=nearest:n_threads=8 -f null -

FFMetric makes a call like this: ffmpeg -hide_banner -nostdin -r 25 -i "filepath2" -r 25 -i "filepath1" -lavfi "[0:v]settb=AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=AVTB,setpts=PTS-STARTPTS[ref];[main][ref]libvmaf=eof_action=endall:log_fmt=json:log_path='C\:/Users/A/AppData/Local/Temp/FFMetrics/db2f3e36-90a1-4d80-a169-5d20264015eb.log':n_threads=7:pool=Harmonic Mean:model=version=vmaf_v0.6.1" -f null -

So i tried re-adding -r 25 before inputs of the ab-av1 ffmpeg call and the vmaf result is the same of the previous versions: 97.805618

But I don't know if this is really the correct case. I will look at the frames manually to see if the quality is really good.

[alexheretic]

Interesting as we used to use -r 24 until that pr. Do you mind sharing the ffprobe, or part of it, for the two videos?

I think we removed it because the justifications for it were dubious. But if we have a good case for it maybe we can restore it, or add it as an option etc.