Problem with BRISQUE scores

[ercanburak]

Bug Description The current implementation of BRISQUE seems erroneous. I encounter negative BRISQUE scores with my data. Upon inspection, I discovered that this behavior is due to #313. When I change the code to use F.interpolate instead of imresize (to revert the change from #313), I no longer encounter negative BRISQUE scores.

To Reproduce Here I share an example image and example codes to reproduce the issue.

• An example image for reproducing the issue:

• A minimal python code:

  
  import cv2
  import torch
  from piq import brisque

img = cv2.imread("brisque_test_img.png", -1) # Read grayscale image img = img / 255.0 # Convert from 0-255 to 0-1 img_tensor = torch.tensor(img) # Convert to torch tensor img_tensor = torch.unsqueeze(img_tensor, 0) # Add third dimension (channel dimension) img_tensor = torch.unsqueeze(img_tensor, 0) # Add fourth dimension (batch dimension) score = brisque(img_tensor)

print("BRISQUE score for the image is {:.4f}".format(score))

- A minimal MATLAB code:
```matlab
img = imread('brisque_test_img.png');
score = brisque(img);
fprintf('BRISQUE score for the image is %0.4f.\n', score)

Here I share the outputs I got:

• Output of python code with latest piq:

  BRISQUE score for the image is -1.1156

• Output of python code when F.interpolate used in piq:

  BRISQUE score for the image is 48.5084

• Output of MATLAB code:

  BRISQUE score for the image is 27.0257

Expected behavior Generate non-negative BRISQUE scores, preferably the same scores as MATLAB.

[zakajd]

Thanks for finding this. @denproc can you take a look? I believe you were the one how implemented BRISQUE and #313

[denproc]

Hi @ercanburak and @zakajd, Thank you for the feedback, good catch! I was able to reproduce the bug. I'll let you know as I gather more information about his weird behaviour.

[ercanburak]

Hi @denproc, do you have any updates on this?

[denproc]

Hi @ercanburak,

Yep, I was just about to write what is going on. Let me first point out, that BRISQUE computes the NSS features, which are used to evaluate the final score by a pre-trained SVR model. While the computation of the NSS features is fixed and does not change across the implementation, the SVR model parameters may vary as one could retrain the SVR model (fitbrisque using built-in matlab function).

Currently, there are two matlab-based implementations, which produce different scores.

• original BRISQUE code with SVR weights released with the paper
• built-in BRISQUE released as a part of MATLAB in 2017.

These two implementations use different SVR parameters

• the original one is trained on LIVE1 dataset producing 774 supporting vectors
• the built-in version uses 593 supporting vectors.

Considering your case, the original matlab code from the authors produces the same negative value estimated by PIQ implementation (see screenshot). It means that our code works as expected, delivering the same results as reported in the original work. However, I see a potential improvement here adding optional weights to be used for SVR, which allow a user to use custom model parameters and match other implementations if needed.

[denproc]

Hi @ercanburak,

After further inspection, we found out that the built-in MATLAB implementation has several dissimilarities compared to the original author's implementation. MATLAB's implementation:

• uses a different set of supporting vectors, coefficients (mentioned in the previous message) and scaling,
• uses replicate mode of padding for 2D convolutions,
• produces slightly different values for NSS features,
• does not use explicit normalisation of features.

As a result, supplying custom parameters for SVM is not enough to match the MATLAB implementation. While access to several implementations might be a sweet feature to have, we focus to deliver reliable implementations, which follow the original methods proposed by authors. Therefore, I'm closing this issue with this message for now. However, you are more than welcome to contribute to the package with a new enhancement issue and/or PR. Also, feel free to reopen the current issue in case of any new updates.

Cheers!

[ercanburak]

Hi @denproc, thank you very much for the detailed explanations!