pycvvdp: Introduce saving quality metrics by default

[vibhoothi]

This allows us to have a CSV file (can be extended to JSON/XML) in future for quality metrics computed by the program. By default it saves to $output/${basename}-metrics.csv format.

This is enabled by default because why not

Edit:

Sample:

Frame Number,cvvdp,PU21-PSNR-Y,PU21-PSNR-RGB2020
0,9.959638,39.131298,38.098446
1,9.880101,38.201233,37.380569
2,9.859364,37.959854,37.151367
3,9.830442,37.814854,37.006905
4,9.864499,37.908867,37.078354
5,9.826247,37.806644,37.008781

[mantiuk]

I know that VMAF can output the quality per-frame and the final score is the mean of those.

This is not the case for cvvdp - it computes the normalized L2 norm of per-frame scores in a different space (before regressing JOD). Therefore, I would be against outputting per-frame scores for cvvdp.

But we will look into creating a text file with the results. Right now, --quiet option could be an alternative.

[vibhoothi]

Thanks,

Are you suggesting that mean of all JODs for a given video is not accurate? Cause that is essentially we get when we use quiet. It is more or less csv of quiet option output in a slightly more readable form

[mantiuk]

The JOD score for video should be computed by running on video files, not individual frames.

I suggest combining frames into video files (with ffmpeg), using lossless compression or no compression.

It is also important to have correct frame-per-second metadata in the video file.

[vibhoothi]

I was trying to load linear light EXR files which is RGB image sequence with correct fps, wouldn’t that be same as loading video as we have frame array there in operation pipeline?

so source and distortion are exr image sequences which is obtained with standard HDRTools library with appropriate conversion files. (From PQ BT2020 sources and compression with codecs)

I strongly believe handling BT.2020 with libavfilter is not a good way to retain original colours since it is inherently not well designed at the moment.

[vibhoothi]

Just to add that initially i was trying to load lossless Y4M/YUV, but tool did not like it https://github.com/gfxdisp/ColorVideoVDP/issues/6

[mantiuk]

cvvdp command line currently does not have direct support for what you are trying to do (process EXR frames as video). However, the correct way of handling frames stored in EXR files is to create a subclass of video_source (pycvvdp/video_source.py) and implement the abstract methods. Then, you can call predict_video_source method (https://github.com/gfxdisp/ColorVideoVDP/blob/7e93a510c79c2dce02772368227d3027e3633ebf/pycvvdp/cvvdp_metric.py#L283) from your own Python code for running the metric on the sequences you have (see the examples folder).

[cosmin]

I know that VMAF can output the quality per-frame and the final score is the mean of those.

This is not the case for cvvdp - it computes the normalized L2 norm of per-frame scores in a different space (before regressing JOD). Therefore, I would be against outputting per-frame scores for cvvdp.

I think it can still be useful to get per frame scores to understand the distribution of quality across the sequence, even if the overall quality is not the average of the frame quality.

For example I'm working on some encoder tuning right now where I get virtually the same cvvdp score on two sequences, but in one case the quality is worse at the beginning and getting better faster and in the other case the quality is more consistently average throughout. But I need to eyeball the heatmap and distograms to understand this, which is less than ideal.

[mantiuk]

You could dump the values that are used to plot a distogram:

https://github.com/gfxdisp/ColorVideoVDP/blob/757f200f0f3116def5c313d654f987feae8931cc/pycvvdp/cvvdp_metric.py#L1071

There is also an experimental feature of dumping channels as video in the branch dump_channels.

What you need is rather specialized and needed for your particular application so I suggest that you customise cvvdp.

If we output per-frame scores, most people will misinterpret those and start computing the mean.