search

chaoma99 / sr-metric

Learning a No-Reference Quality Metric for Single-Image Super-Rolution
https://sites.google.com/site/chaoma99/sr-metric
MIT License
132 stars 36 forks source link

inconsistent result with human perception #1

Open ceciliavision opened 5 years ago

ceciliavision commented 5 years ago

Hello,

I ran the code to compare the following two images' perceptual score. However, I got results that hardly make sense, and I hope to get some insights from you.

Here are the two images:

Image A: image

Image B: image

Image A gets a score of 7.2, while image B gets a score 8.9. This is not consistent to what they appear to our human perception. I'm wondering if there is certain bias in the learned metric, that fails on the images I presented here?

To make it more complete, the way I tested these is to simply run: img = imread(path); score = quality_predict(img);

Is there additional processing that's required to run the metric correctly?

Thanks,

Vandermode commented 5 years ago

Hi, the issue described above is due to the IQA model (in this repo) can only assess quality degradations arising from the distortion types (i.e. super-resolved artifacts) that it has been trained on.

Such model could be formulated as 'distortion aware', thus is necessarily limited. For more general purpose 'distortion unaware' NR IQA, please refer to the NIQE [1] to meet your goal.

Hope this helps.

By the way, I will close my issue in your repo since I have figured out everything mentioned there. Thanks for your previous insightful discussion.

[1] Mittal, Anish, Rajiv Soundararajan, and Alan C. Bovik. "Making a" Completely Blind" Image Quality Analyzer." IEEE Signal Process. Lett. 20.3 (2013): 209-212.

ceciliavision commented 5 years ago

Hi, the issue described above is due to the IQA model (in this repo) can only assess quality degradations arising from the distortion types (i.e. super-resolved artifacts) that it has been trained on.

Such model could be formulated as 'distortion aware', thus is necessarily limited. For more general purpose 'distortion unaware' NR IQA, please refer to the NIQE [1] to meet your goal.

Hope this helps.

By the way, I will close my issue in your repo since I have figured out everything mentioned there. Thanks for your previous insightful discussion.

[1] Mittal, Anish, Rajiv Soundararajan, and Alan C. Bovik. "Making a" Completely Blind" Image Quality Analyzer." IEEE Signal Process. Lett. 20.3 (2013): 209-212.

Thanks for letting me know.

Regarding the answer to perceptual metric, what I showed above is actually a result from a super-resolution model. I agree there would be certain limitation, but was trying to get an intuitive explanation about that limitation (e.g. whether high frequency details are generally considered as 'good images').