How to generate CAMBI heat maps?

[ltrudeau-twoorioles]

Is it possible to dump the CAMBI confidence data using the VMAF executable as is done in the tech blog?

With this data, heat maps can easily be generated using either Matlab or Matplotlib (in a Jupyter notebook).

[kylophone]

cc @jsole, @nilfm, @mafonsoN

Do you already have a buffer like this inside the feature extractor? If so, it might be good to add an option to dump the buffer to disk. Maybe use PGM?

[ltrudeau-twoorioles]

Looks like the data is in the c_values buffer see https://github.com/Netflix/vmaf/blob/master/libvmaf/src/feature/cambi.c#L272

update: Looks like c_values are floats so it might require fixed-type conversion or PFM

[nilfm]

I agree it's a good idea to add a flag to dump this to disk. I would think using PFM is better so the rescaling is simpler. From what I've seen, the values in PFM need to be between 0 and 1 so we could divide by the maximum theoretical c_value for a pixel. This value, according to my calculations, can be computed as max(diff_weights) * window_size^2 / 4, and for the current default parameters is a bit under 4000.

Interested in what others have to say, it's probably worth trying and then tuning the normalization parameter if it seems too large.

[kylophone]

I agree it's a good idea to add a flag to dump this to disk. I would think using PFM is better so the rescaling is simpler. From what I've seen, the values in PFM need to be between 0 and 1 so we could divide by the maximum theoretical c_value for a pixel. This value, according to my calculations, can be computed as max(diff_weights) * window_size^2 / 4, and for the current default parameters is a bit under 4000.

Interested in what others have to say, it's probably worth trying and then tuning the normalization parameter if it seems too large.

Try it out, hopefully not too much work. As for putting an fopen and fwrite inside of a feature extractor, it might be worth having all of this behind some #ifdefs.

[nilfm]

Hi @ltrudeau-twoorioles, the branch feature/nilfm-cambi-heatmaps implements this feature, pending some testing and checking cross-platform compatibility. You can try it and see if it works for your purposes by adding the flag heatmaps_path=some_path to CAMBI. Some considerations:

• The path some_path has to point to an already-existing directory.
• This will generate one .pfm file for each frame and scale. We might want to change this to generate a single .yuv file per scale, which would be an easy change and allow us to see the temporal variation of bands, but it isn't implemented as of now.
• The .pfm images might be vertically flipped depending on which reader you are using. I am not sure why different readers seem to behave differently in this regard.
• Having the heatmaps_path flag active slows the program down by about 2x in my experience. If the flag isn't used, by default it doesn't write anything to disk.

Here is (a screenshot of) one of the maps generated by this:

Please let me know if you have any questions/suggestions, and contributions are welcome! :)

[ltrudeau-twoorioles]

That's some great work @nilfm, thanks for the branch! Works great for me on MacOS. I opened the .pfm with Preview and it wasn't flipped.

Scale 0:

The hope is that this map would help better understand how different parts of the frame contribute to the overall CAMBI score. For example, banding in dark vs brighter regions or how CAMBI react to film grain.

[kylophone]

@nilfm, are you going to make a PR for this?

[nilfm]

@kylophone, I made the changes to output a 16-bit YUV400 (grayscale) video for each scale instead of separate frames, and I'll put the PR up now. I think it helps visualize the temporal movement of the bands, and it should be easy to take a single frame from it if necessary. We can discuss futher on the PR, and we can always not push this last commit if we want the old behaviour instead.

Edit: https://github.com/Netflix/vmaf/pull/947 is the PR.

[kylophone]

Closed, since #947 is now merged.