2 ffmpeg's, 2 VMAF scores (same videos)? / Pre-scored test videos?

[AlexMhr]

Running the command on the same two files:

ffmpeg.exe -i "Distorted.mkv" -i "Reference.mkv" -filter_complex "libvmaf=model=version=vmaf_v0.6.1" -f null -

first using ffmpeg 5.1.2: https://www.gyan.dev/ffmpeg/builds/packages/ffmpeg-5.1.2-full_build.7z

results in

VMAF score: 94.845854

but

using ffmpeg 6.0: https://www.gyan.dev/ffmpeg/builds/packages/ffmpeg-6.0-full_build.7z

results in

VMAF score: 93.693399

I've zipped the two videos, two ff logs, two vmaf logs here: VMAF.zip and here (redundant): https://ufile.io/anvq589s

Any help would be appreciated, I'm afraid this is past me.

Not to mention clarity - which score is correct?

---

I've also been scouring the web for any pre-scored distorted/reference video samples that I might re-test with my differing ffmpeg versions in order to see which matches the official vmaf score (which is the correct score).

Do any such files exist?

Thank you. A

[krasuluk]

Hi Alex,

I'm not entirely sure what is causing the discrepancy between the two versions of ffmpeg (I'll let @kylophone comment on that).

As for some videos with results, I think the Python Library tests could provide what you are looking for. You can checkout the test cases, for example, in this file.

The destination and other useful information about the videos used by those test cases can be found here. Is that what you've been looking for?

[AlexMhr]

Thank you, yes that's it, I really appreciate that.

So long story short, I converted a .yuv reference/distorted pair to FFV1 yuv420p10le and yuv420p (4 files total).

Then I ran the vmaf command on all the combinations through both FF 5.1 and 6.0, and the results are:

FF 5.1 | Ref 08 Bit | Dis 08 Bit | VMAF score: 76.668905 FF 6.0 | Ref 08 Bit | Dis 08 Bit | VMAF score: 76.668905 FF 5.1 | Ref 10 Bit | Dis 08 Bit | VMAF score: 76.668905 FF 6.0 | Ref 10 Bit | Dis 08 Bit | VMAF score: 76.668905 FF 5.1 | Ref 10 Bit | Dis 10 Bit | VMAF score: 74.715529 FF 6.0 | Ref 10 Bit | Dis 10 Bit | VMAF score: 76.668905 FF 5.1 | Ref 08 Bit | Dis 10 Bit | VMAF score: 74.715529 FF 6.0 | Ref 08 Bit | Dis 10 Bit | VMAF score: 76.668905

So it would seem it's FF 5.1 that diverges from the pack when the distorted file is 10 bit. Everything else is consistent.

FWIW, running vmaf directly on the .yuv files without converting to FFV1 also scored 76.668905 (both FF6 & FF5).

So unless I'm mistaken, it looks like a bug has already been fixed, and FF 6 is the trusted build.

That's my takeaway at this moment anyway.

But if you think I've missed something by all means let me know.

Thanks again for the files, A