search

kornelski / dssim

Image similarity comparison simulating human perception (multiscale SSIM in Rust)
https://kornel.ski/dssim
GNU Affero General Public License v3.0
1.09k stars 70 forks source link

Consider evaluating DSSIM using CLIC-2021 peceptual quality task #102

Open jyrkialakuijala opened 3 years ago

jyrkialakuijala commented 3 years ago

See the similar issue in libjxl.

It would be great to evaluate DSSIM on the CLIC-2021 perceptual quality task. This should provide additional information to he community with respect to its performance characteristics when compared to other potentially usable perceptual quality metrics.

Please use the test data from the CLIC 2021 perceptual challenge to generate a CSV file with the decisions (see link below for the exact instructions):

https://github.com/fab-jul/clic2021-devkit/blob/main/README.md#perceptual-challenge

Please email them to George Toderici (or me) to get the final results / ranks. We'll publish these at: http://compression.cc/leaderboard/perceptual/test/

...and of course update this bug tracker.

kornelski commented 3 years ago

Test set reports accuracy 0.616, which is just slightly above the ms-ssim submission.

kornelski commented 3 years ago

@jyrkialakuijala Here's the full submission dssim3.csv.zip

jyrkialakuijala commented 3 years ago

Do you know if there is a way to compute three different accuracy/correlation values, one for raters testing the 0.07 bpp images, one for 0.15 bpp images and yet another for 0.3 bpp?

Motivation for this idea: I would expect different correlations for these. Also, I'd expect classic methods to work better for higher bitrates and possibly neural image quality methods to work better at the low.

kornelski commented 3 years ago

The small test set and the tool that is public don't use the actual scoring method, and don't expose any more info about the images:

https://github.com/fab-jul/clic2021-devkit/issues/4

kornelski commented 3 years ago

I was unable to submit at http://compression.cc — the site very frequently times out. I think I've barely registered after a dozen of tries, but I can't log in, because the login form asks for username, and the registration didn't register any username. I've tried using e-mail as the username, but it didn't work.

gtoderici commented 3 years ago

Hi @kornelski - I am one of the CLIC organizers. I don't think the registration is opened anymore. We'll update to reflect this. With that said, please send me the submission file (i.e., the test set CSVs), and I'll be able to update this thread with the numbers. I have the email username@google.com (where username is gtoderici).