CompareAllExampleScenes() difference image summary statistics

[benjamin-heasly]

CompareAllExampleScenes() compares multi-spectral images that were generated at different times, or on different machines. This helps validate code and configurations.

Currently, the function creates multispectral difference images and reports the min and max pixel value from each difference image. When plotted, the min and max statistics sometimes pop out, facilitating detection of incorrect renderings.

But we probably need more statistics besides min and max. For example, I found that the PBRT anisoward material was broken on a particular machine. I had one machine that rendered the anisoward material correctly, while the broken machine rendered a default matte material. The renderings look very different by eye, and the difference image has a giant hole in it. But the max pixel difference was only on the order of 2.

In contrast, some scenes have uncorrelated, harmless noise that is the expected result of stochastic ray sampling. Some of these difference images have max pixel differences that are much larger than 2.

So sometimes a correct rendering pops out during comparison, where an incorrect rendering does not. We should think of additional comparison statistics that will capture an overall sense of image sanity or sameness, as opposed to per-pixel differences.

Maybe a mean-absolute-pixel-difference would help.

[benjamin-heasly]

One problem with mean-absolute-pixel-difference is that images with black backgrounds will almost always have a mean difference near 0. The black is easy to render consistently, so there will be lots and lots of difference pixels with a value close to 0

[benjamin-heasly]

In 527e4cb28889d20cd6c34276682ff1d02fbc45ab, CompareAllExampleScenes() reports correlation coefficients of multi-spectral pixel components, rather than difference image mins and maxes. This seems to capture a better sense of "sameness" between multi-spectral renderings.

[benjamin-heasly]

David suggests:

Another useful measure might be the maximum percentage difference, with max taken over all pixels with a value greater than some threshold, perhaps 0.2. The visual system is sensitive to percentage differences, which is why this is sensible. Setting the threshold is needed because the numbers can blow up when the denominator is small. (This is also true of the statement the visual system is sensitive to percentage differences -- it's an approximation that holds once the baseline level rises above some threshold.)

[benjamin-heasly]

8cef3e7a80b1198a9c9d6f7cfa87b9f35a35fe0d adds a proportional difference statistic as David suggested. Plots min and max proportional differneces as well as correlation coefficients.