Query regarding reproducuibilty of MSU Benchmarks

[kanlions]

MSU Benchmarks report results in the form of vmaf 061 YUV 611, vmaf YUV 062 411. Can anyone here provide the exact way the results are calculated and exact models used as I see multiple models in the model folder with similar configurations. Any help or documentation in this regard will be very helpful.

[nilfm99]

Hi, those benchmarks were performed by MSU independently of Netflix and according to their website, VMAF was obtained through their tool MSU VQMT. We don't have insights into what settings they used.

[kanlions]

Thanks for your reply. One more query is it possible to get VMAF Y, U,V separately from command line interface or code so that manually I can calculate 10:1:1, 6:1:1 etc.

[nilfm99]

VMAF is designed to be run on the luma (Y) channel, and I'm not aware of any guarantee of accuracy if run on the chroma channels independently. From a quick look at their website, though, it does seem like they did that. If you're looking to reproduce their results, you can try ffmpeg's extractplanes filter to obtain the {Y, U, V} planes as single-channel raw video and run libvmaf on those, taking care of supplying the correct width and height depending on the chroma subsampling.

[kanlions]

Thanks a lot for your reply. Even my understanding of VMAF from the feature extraction steps was that it was for Y channel only. SInce the models you provide like 0.6.1 are SVM (ML) based so I presume trained only on Y channel data from the videos. So I decided to ask here since you are the creators of VMAF. If I remember I attended EI once and Lukas mentioned samething.

Recently the same group reported their dataset in NIPS dataset track and in the result they report VMAF Y and VMAF YUV separtely so I thought of investigating further.

Thanks for the help. SInce you mentioned ffmpeg I have an unresolved issue in ffmpeg with vmaf [AVFilterGraph @ 0x55f463753b40] No such filter: 'libvmaf'

I followed all the instructions here https://github.com/Netflix/vmaf/blob/master/resource/doc/ffmpeg.md But still it seems not to show that error. Can you please give any suggestion for this.

[nilfm99]

No worries, thanks for bringing this to our attention as well.

Regarding the ffmpeg-libvmaf failure, you can either:

1. Install a statically linked ffmpeg build that brings libvmaf with it: for example, this one should have it. This one has an older libvmaf version, 2.3.0, so it may not work for you if you want the latest build.
2. Build libvmaf and ffmpeg yourself, following first the instructions to build and install libvmaf, and then the instructions to install ffmpeg.

Let me know if that doesn't work or if the docs need clarifications, if so, please also tell me what OS you are using.

[kanlions]

In case your team is interested these are the findings of MSU https://proceedings.neurips.cc/paper_files/paper/2022/file/59ac9f01ea2f701310f3d42037546e4a-Paper-Datasets_and_Benchmarks.pdf https://proceedings.neurips.cc/paper_files/paper/2022/file/59ac9f01ea2f701310f3d42037546e4a-Supplemental-Datasets_and_Benchmarks.pdf

---

Regarding FFmpeg OS Ubuntu 22.04 VMAF version 4c08f00b vmaf_v0.6.1: 96.302174 which -a ffmpeg /usr/local/bin/ffmpeg I have also added the path in bashrc.

But I dont know what is going wrong in the installation process as the command line vmaf is working so I guess libvmaf must be installed correctly and ffmpeg is built from instructions so is there any upgraded or downgraded version of ffmpeg I should use

Thanks for all the support till now.

[nilfm99]

I built the latest master branch of both libvmaf and ffmpeg on a fresh Ubuntu 22.04 machine to make sure we're not missing anything, this is what I did:

git clone https://github.com/Netflix/vmaf.git
cd vmaf
python3 -m venv .venv
source .venv/bin/activate
pip install meson ninja cython numpy
sudo apt install -y nasm build-essential python3-dev
make install
cd ..

git clone https://github.com/FFmpeg/FFmpeg.git
cd FFmpeg
./configure --enable-libvmaf
make -j4
sudo ldconfig
cd ..

FFmpeg/ffmpeg_g -i dis.y4m -i ref.y4m -lavfi libvmaf -f null -

If this still doesn't work, please let me know the output of the following commands:

ldd FFmpeg/ffmpeg_g

for me, it's the following:

linux-vdso.so.1 (0x00007ffd349b2000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f96a8dfa000)
        libvmaf.so.3 => /usr/local/lib/x86_64-linux-gnu/libvmaf.so.3 (0x00007f96a8cdc000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f96a8cc0000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f96a8a95000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f96aba75000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f96a8869000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f96a8847000)

and

FFmpeg/ffmpeg_g -filters | grep vmaf

for me, it is:

ffmpeg version N-114123-gb3d87fb0c4 Copyright (c) 2000-2024 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.4.0-1ubuntu1~22.04)
  configuration: --enable-libvmaf
  libavutil      59.  1.100 / 59.  1.100
  libavcodec     61.  1.100 / 61.  1.100
  libavformat    61.  0.100 / 61.  0.100
  libavdevice    61.  0.100 / 61.  0.100
  libavfilter    10.  0.100 / 10.  0.100
  libswscale      8.  0.100 /  8.  0.100
  libswresample   5.  0.100 /  5.  0.100
 ... libvmaf           VV->V      Calculate the VMAF between two video streams.
 ... vmafmotion        V->V       Calculate the VMAF Motion score.

[kanlions]

Thanks a ton for all the support. It worked now. May be earlier built would have been corrupted.

[nilfm99]

Awesome, glad it worked! Closing this thread, please reopen or open a new issue if you have further problems.