Could you provide the code for the test pipeline based on the given checkpoint ?

[clearlovewl]

Hello YangiD,

Thank you very much for sharing your work and providing the checkpoint model. I have tried testing the model using the provided checkpoint with the following code:

    def test(self, data):
        """Testing"""
        self.model_hyper.train(False)
        pred_scores = []
        gt_scores = []
        model = HyperIQA(r"D:\doingproject\DefenseIQA-NT\checkpoints\livec_bs16_grad[1]_weight[0.001]_h[0.01].pth").to("cuda")
        for img, label in data:
            # Data.
            img = torch.tensor(img.cuda())
            img,_,_=IFGSM_IQA_untarget(model,img,label,eps=0.03, alpha=0.01, iteration=1)
            label = torch.tensor(label.cuda())

            paras = self.model_hyper(img)
            model_target = hyper.TargetNet(paras).cuda()
            model_target.train(False)
            pred = model_target(paras['target_in_vec'])

            pred_scores.append(float(pred.item()))
            gt_scores = gt_scores + label.cpu().tolist()

        pred_scores = np.mean(np.reshape(np.array(pred_scores), (-1, self.test_patch_num)), axis=1)
        gt_scores = np.mean(np.reshape(np.array(gt_scores), (-1, self.test_patch_num)), axis=1)
        test_srcc, _ = stats.spearmanr(pred_scores, gt_scores)
        test_plcc, _ = stats.pearsonr(pred_scores, gt_scores)

        self.model_hyper.train(True)

        print( test_srcc, test_plcc)

        return test_srcc, test_plcc

However, I encountered an issue as the final results were not as expected. Could you please let me know if there is anything I might have missed or if there are specific requirements for reproducing the results?

Any guidance or suggestions would be greatly appreciated.

Thank you for your time and support!

Best regards, clearlovewl

[YangiD]

Hello, thank you for your attention to our work! In the code you provide, there are two possible problems:

1. param.requires_grad = False is ignored and needs to be added after model_target.train(False) :

   for param in model_target.parameters():
     param.requires_grad = False

2. In your FGSM, the input score is the label instead of the predicted score of the original image.

Here is a part of our test code that we hope will help you:

for l in range(len(data)):
    datai = data[l]
    imgname = datai['filename']
    ori_imhi = None
    imgi = None

    for i in range(25):
        imgpath = os.path.join(test_dir,imgname[:-4]+'_'+str(i)+'.bmp') #imgname[-4:] '.bmp' '_20.bmp'
        img = pil_loader(imgpath)
        img = transforms(img)
        img = torch.tensor(img.cuda()).unsqueeze(0)

        oriimgpath = os.path.join(test_ori_dir,imgname[:-4]+'_'+str(i)+imgname[-4:])
        ori_img = pil_loader(oriimgpath)
        ori_img_np = np.array(ori_img).astype(np.float)
        ori_img = transforms(ori_img)
        ori_img = torch.tensor(ori_img.cuda()).unsqueeze(0).cuda()

        if imgi is None:
            imgi = img
            ori_imgi = ori_img
        else:
            imgi = torch.cat((imgi,img),dim=0) #torch.Size([25, 3, 224, 224])
            ori_imgi = torch.cat((ori_imgi,ori_img),dim=0)

    paras = model_hyper(imgi)  # 'paras' contains the network weights conveyed to target network

    # Building target network
    model_target = models.TargetNet(paras).cuda()
    for param in model_target.parameters():
        param.requires_grad = False

    # Quality prediction
    pred = model_target(paras['target_in_vec'])

    scorei = pred.detach().cpu().numpy()

    scorei = scorei.astype(np.float) #(25,)
    scorei = scorei.mean() #()
    pred_scores.append(scorei)

    ori_paras = model_hyper(ori_imgi)
    model_target_ori = models.TargetNet(ori_paras).cuda()
    for param in model_target_ori.parameters():
        param.requires_grad = False
    ori_pred = model_target_ori(ori_paras['target_in_vec'])
    ori_scorei = ori_pred.detach().cpu().numpy()
    ori_scorei = ori_scorei.astype(np.float)
    ori_scorei = ori_scorei.mean()
    pred_scores_ori.append(ori_scorei)

    mosi = datai['mos']
    moses.append(float(mosi))

pred_scores = np.array(pred_scores).squeeze()
moses = np.array(moses).squeeze()
pred_scores_ori = np.array(pred_scores_ori).squeeze()

mos_max = 92.43195
mos_min = 3.42
pred_scores = (pred_scores-mos_min)/(mos_max-mos_min)*100
pred_scores_ori = (pred_scores_ori-mos_min)/(mos_max-mos_min)*100
moses = (moses-mos_min)/(mos_max-mos_min)*100

rho_s, _ = spearmanr(pred_scores, moses)
rho_p, _ = pearsonr(pred_scores, moses)
rho_k, _ = kendalltau(pred_scores, moses)
rmse = np.sqrt(np.mean(np.power((pred_scores-moses),2)))
print('Attack: SROCC/PLCC/KROCC/RMSE\n{0:.4f}\t{1:.4f}\t{2:.4f}\t{3:.4f}'.format(rho_s,rho_p,rho_k,rmse))

[clearlovewl]

Thank you for your response!

After modifying the attack strategy by using the predicted score instead of the true label, and the crop image score instead of the mean image score, I observed a significant performance improvement.