search

Netflix / vmaf

Perceptual video quality assessment based on multi-method fusion.
Other
4.68k stars 757 forks source link

Wrong PSNR and VMAF values in FFmpeg 5.1.1 #1109

Open waveletbeam opened 2 years ago

waveletbeam commented 2 years ago

We recognized that adding our Wavelet Beam Logo to a video stream didn’t result in a drop of PSNR, when using the PSNR calculation in the FFmpeg VMAF filter. Additionally PSNR is limited at 72dB for 10bpc and to 60 dB for 8bpc (#371 and #1115). This is wrong! "Quantisation noise of the digitisation process" does not matter in the further workflow steps. For distribution encodes we cannot reach this high PSNR values but if you like to use PSNR for other workflow steps, higher PSNR values are important.

Reference video UHD FFv1: black screen with no Wavelet Beam logo: http://projekte.waveletbeam.com/black.mkv

Video under test: black screen with Wavelet Beam logo: http://projekte.waveletbeam.com/blackLogo.mkv

This FFMPEG command gives us the correct PSNR value of 28dB: ffmpeg.exe -i G:\TestFFMPEG5\blackLogo.mkv -i G:\TestFFMPEG5\black.mkv -filter_complex "psnr" -f null -

The VMAF filter with PSNR enabled gives us a PSNR value of 72bB and VMAF of 100: ffmpeg -i G:\TestFFMPEG5\blackLogo.mkv -i G:\TestFFMPEG5\black.mkv -filter_complex "libvmaf=feature=name=psnr:model=version=vmaf_4k_v0.6.1:log_fmt=xml:log_path="G\:/TestFFMPEG5/black2.xml" " -f null -

We think that 72dB but at the same time 100 VMAF points are wrong results as the difference in the two videos can be easily recognized. Additionally, the PSNR value of 72dB is mathematically wrong.

kylophone commented 2 years ago

It is very possible that you have a PTS misalignment, you can try using setpts to align the streams as explained here: https://github.com/Netflix/vmaf/blob/master/resource/doc/ffmpeg.md. To rule out a libvmaf frame mismatch, you can try these sources with the vmaf command-line tool.

waveletbeam commented 2 years ago

Thank you for the advice but it’s not a pts mismatch problem. A pts mismatch should not be a problem for the PSNR/VMAF calculation because the image content of each frame is the same. For the reference video (black.mkv) all pixels of all frames are black (0). For the tested video blackLogo.mkv we only have additionally the Wavelet Beam logo in all frames.

richardpl commented 2 years ago

See #1115