Open zakajd opened 3 years ago
Thanks for raising the issue @zakajd. This is interesting and seems worth having.
To maintain backwards compatibility, I would suggest implementing it via a new keyword-only argument, automatic_downsampling
(or auto_downsample
), that defaults to False.
Description
They're 2 MATLAB implementations of SSIM from original authors, which can be found here. First one
ssim_index.m
matches current implementation inskimage.metrics
. Second onessim.m
differs by image preprocessing. Before computation of SSIM authors propose to perform average pooling with factorF
, whereF = max(1, round(min(H, W)/256))
.Those two implementation are equivalent for small images (with min size < 384), so why matter? SSIM and other full-reference quality assessment metrics (PSNR) are often used as "perceptual simularity estimators", meaning that you get bigger values for more "similar" images. Quality of metrics predictions is benchmarked on a set of human labeled images (Mean Opinion Scores databases) by Spearman Rank Order Correlation (SRCC) and Kendal Rank Order Correlation (KRCC). Higher correlation means that metric predictions of "distance" between images is consistent with human judgements.
I've run test to compare performance on TID2013 database which is often used in image quality assessment papers. Here is the results:
So current code measures "perceptual simularity" even worse than simple MSE between images. 😞 Exact reasons for such behaviour is not the scope of this issue
Way to reproduce
Code I've used for computation is a bit messy. It's based on https://github.com/photosynthesis-team/piq/blob/feature/benchmark/tests/results_benchmark.py with additional wrapper for
skimage
to support tensors as inputs.How to fix
Simple way to match the performance is add 4 lines of code before beginning of SSIM computation https://github.com/scikit-image/scikit-image/blob/8acad22ff31d44b17651003db49791640d9b0b41/skimage/metrics/_structural_similarity.py#L187-L189
Version information
I can open PR if you interested