on resizing operation

[ciwei123]

@lidq92 Thanks for your sharing. I test the model p1q2plus0.1variant.pth you provided, the result is:

/LinearityIQA-master_test/test_dataset.py --dataset CLIVE --arch resnext101_32x8d --trained_model_file /mnt/fhl/LinearityIQA-master/p1q2plus0.1variant.pth
Namespace(P6=1, P7=1, alpha=[1, 0], angle=2, arch='resnext101_32x8d', augment=False, batch_size=8, beta=[0.1, 0.1, 1], crop_size_h=498, crop_size_w=498, data_info={'KonIQ-10k': './data/KonIQ-10kinfo.mat', 'CLIVE': './data/CLIVEinfo.mat'}, dataset='CLIVE', epochs=30, exp_id=0, ft_lr_ratio=0.1, hflip_p=0.5, im_dirs={'KonIQ-10k': '/data_sdb/fhl/KonIQ-10k/images/', 'CLIVE': '/data_sdb/fhl/CLIVE/Images/'}, loss_type='norm-in-norm', lr=0.0001, p=1, pool='avg', q=2, resize=False, resize_size_h=498, resize_size_w=664, save_result_file='results/dataset=CLIVE-tested_on_p1q2plus0.1variant.pth', seed=19920517, train_and_val_ratio=0, train_ratio=0, trained_model_file='/mnt/fhl/LinearityIQA-master/p1q2plus0.1variant.pth', use_bn_end=False)
# test images: 1162
CLIVE, SROCC: 0.797
CLIVE, PLCC: 0.832
CLIVE, RMSE: 11.731
CLIVE, SROCC1: 0.795
CLIVE, PLCC1: 0.827
CLIVE, RMSE1: 11.856
CLIVE, SROCC2: 0.795
CLIVE, PLCC2: 0.830
CLIVE, RMSE2: 11.571

The SROCC is 0.797， is inconsistent with yours(0.834).

[lidq92]

@ciwei123 Please follow README, you need to specify --resize.

[ciwei123]

@ciwei123 Please follow README, you need to specify --resize.

@lidq92 Thanks for your reply. I get the same result with yours.when I add the command --resize. And I find that the method of read the image and preprocess should be same with the train phase.

[ciwei123]

@lidq92 I train my model by changing some layers.My models are obtained in the same setting with you.

1. I find that although I resize the image during train phase, some results with resize are worse than without.
2. In addition, I train my model with the same setting with you(resize 498*664), and test my model on TID2013 dataset, when I resize the image to 500*500is better than resize 498*664. I am confused about the result,could you help me? Thank you very much.

[lidq92]

@ciwei123

1. If you change the model, some hyper-parameters may need to be tuned based on the validation set.
2. There is a trade-off between resizing test images to adapt the training inputs or not resizing them to avoid distortion introduced by resizing.
3. What is your training set? If it is KonIQ-10k, then you would not expect a good result for TID2013 since the distributions between these two datasets are quite different. And for your question, I would expect the result when resizing to 498x664 to be better since the image resolution in TID2013 is 384x512. Your "unexpected" results may come from your model training process. And if both two results are bad, then the distribution discrepancy is the major concern for your question (You need to improve the generalization ability of your model).

[ciwei123]

@lidq92 Thanks for your very quick reply.

1. I just change the Resnet101 to Mobilenet , because I want to get a Light model. I don't change the hyper-parameters, could you tell me which hyper-parameters need to be adjusted？Any suggestions on lightweight are welcome.

There is a trade-off between resizing test images to adapt the training inputs or not resizing them to avoid distortion introduced by resizing.

You means that the settings during training and testing may be different，depending on the resolution of the test data set? my result on TID2013: raw is no resize

500*500/raw/664*498

0.500/0.501/0.489
0.502/0.502/0.493
0.500/0.498/0.494
0.538/0.507/0.502

I think 664*498 should be better, but the 500*500 is better, so I am very confused.

3.My training set is only KonIQ-10k(7058 images),and resize to 664*498. I think only test model on CLIVE and KonIQ-10k is very limited, I want to show the effectiveness of the model on more data sets，so I test the model on TID2013 and LIVE dataset, and I don't finetune the model on TID2013、 LIVE and CLIVE dataset. Could you give me any good suggestions for improving generalization?

[lidq92]

@ciwei123 As I have seen, both two results were bad, and the distribution discrepancy is a major concern.

What is the result of your trained MobileNet model on KonIQ-10k test set? If it is not good, you may re-train the model with other optimization hyper-parameters ( I guess the default values of hyper-parameters may be OK).

You may refer to mixed dataset training or domain adaptation for better overall performance. Good luck.

[ciwei123]

@lidq92 The result of my trained MobileNet model on KonIQ-10k test set is SROCC about 0.900, so the hyper-parameters may be OK.

And different datasets have different ranges of mos，how to deal with this? Is it because it has been normalized（by using norm-in-norm）, so there is no need to consider this? Is it okay when there are images from different data sets in a batch? Thank you very much!

[lidq92]

@ciwei123 You may refer to MDTVSFA and UNIQUE for mixed datasets training.

[ciwei123]

@ciwei123 You may refer to MDTVSFA and UNIQUE for mixed datasets training.

@lidq92 Thank you very much.