jonsneyers
commented
7 months ago SSIMULACRA2, just like DSSIM, is based on MS-SSIM. When implementing MS-SSIM, there are some choices to be made, such as:
- What colorspace to apply it to? Luma only or also look at colors?
- What weights to use for each scale?
- How to aggregate the heatmap into a single number?
Which specific variant of MS-SSIM would you like to have? When I tested the libvmaf version of MS-SSIM, it didn't really show a very good correlation with most IQA datasets: https://github.com/cloudinary/ssimulacra2?tab=readme-ov-file#metric-performance
That MS-SSIM implementation is OK in terms of correlation when looking at the datasets that contain extremely low quality images, like Kadid-10k, but in the more usable/typical range of qualities (say, libjpeg-turbo q40-95), it doesn't really seem to bring any value over the metrics we already have in benchmark_xl.
Not saying we shouldn't add it, just that it's probably a bad idea to optimize for this metric or to trust it to predict image quality well (no metric should really be trusted for that, of course).
MS-SSIM is a popular metric for the team AVIF, could the bit of code necessary to compute it be added to the libjxl repo and the result of the MS-SSIM score be added to the output of a benchmark_xl run ?
This would add a new column, yet gives way to setup adequate comparisons and new arguments to tackle the proponents of video based image formats :)