search

JoshuaEbenezer / ChipQA

Implementation of ChipQA (https://ieeexplore.ieee.org/document/9540785)
https://joshuaebenezer.github.io/publication/chipqa/
9 stars 2 forks source link

Training with CSIQ-VQA database with OptimizeWarning #2

Open IRON-HUGHY opened 1 year ago

IRON-HUGHY commented 1 year ago

Hi Joshua, I tried to use chipQA to train and test its performance on a CSIQ-VQA database, which is a 480p resolution database. 1 - I run chipqa_yuv.py get the feature file 2 - I run python cleaner_svr.py --score_file /path/to/score.csv --feature_folder feature_folder --train_and_test I got the all_pred = [33.22243675 49.20781647 50.0144947 41.65980157 41.57749611 40.87659356, 61.43243688 58.08024139 38.79844636 58.73480296 46.18201971 36.00328378, 36.96726173 42.92148637 53.04071207 63.78199282 53.71268326 31.00923555, 43.38495816 44.25350569 43.77634616 44.97062795 44.97958702 44.92278459, 49.21896745 50.34141739 48.01144443 50.7113197 53.70022118 53.09089083, 36.81422243 42.11125906 47.5120631 48.23488525 52.38174794 59.13939887, 43.95188126 40.72350856 39.07306079 39.36255459 38.9502675 37.87678828, 36.04681885 38.40384619 41.65701426 40.27273715 42.31758531 43.51008817, 33.85725812 39.69753213 48.37801187 43.89804409 44.25391719 44.2835234 ] while all_dmos = [30.96926305 55.12598144 68.47434569 63.19511855 44.82338636 39.05790258, 61.77650141 43.23640628 27.29835247 55.44590755 39.26934316 28.35209766, 29.65262045 41.59899252 55.96629109 71.88924176 34.53450825 17.19991484, 26.41774852 46.88645613 79.15402107 50.65216125 45.56089672 36.8136669, 64.98568555 49.49478196 29.56212976 68.14515278 49.32783463 39.70465263, 29.9807714 40.99972896 58.42660889 85.19789617 53.10270256 24.20572819, 31.1199481 51.57753506 64.18712587 61.9394302 52.56382728 39.07539394, 61.0901873 42.19819949 27.14502109 73.33284815 42.48086625 34.19254771, 26.21587958 40.64126874 52.39742685 70.05419936 42.42780915 26.91805556] I would get "lib\site-packages\scipy\optimize\minpack.py:834: OptimizeWarning: Covariance of the parameters could not be estimated category=OptimizeWarning)" several times during this step. And the results of PLCC and SROCC only (0.33308124176152726, 0.3385172479512102).

Can you tell me how to solve this problem and get the right result.

Thanks in advance, Hugh

JoshuaEbenezer commented 1 year ago

Hi Hugh

The optimization warning comes up when there's no good fit available for the mapping. This can happen when the predictions are very different from the scores, which I suppose is happening here. I've not tested ChipQA on the CSIQ database but I would expect the SRCC to be around 0.7 since BRISQUE and TLVQM get an SRCC of around that level (Table IV in https://arxiv.org/pdf/2206.14713.pdf).

In line 67 of cleanersvr.py, I'm making the assumption that the first element of the '' separated list of the filenames in the dataframe corresponds to the content. If this is not the case, you have to define a 'content' column in the dataframe yourself. Also the code removes distortions marked as 'p', which in our original database corresponded to proprietary content. That might be one reason why performance is poor.

I will update cleaner_svr.py to fix these things, but you may have to make more changes to the SVR code to make it run correctly for CSIQ, depending on how you created your score CSV.

Regards, Josh

On Tue, Feb 14, 2023 at 3:16 AM IRON-HUGHY @.***> wrote:

Hi Joshua, I tried to use chipQA to train and test its performance on a CSIQ-VQA database, which is a 480p resolution database. 1 - I run chipqa_yuv.py get the feature file 2 - I run python cleaner_svr.py --score_file /path/to/score.csv --feature_folder feature_folder --train_and_test I got the all_pred = [33.22243675 49.20781647 50.0144947 41.65980157 41.57749611 40.87659356, 61.43243688 58.08024139 38.79844636 58.73480296 46.18201971 36.00328378, 36.96726173 42.92148637 53.04071207 63.78199282 53.71268326 31.00923555, 43.38495816 44.25350569 43.77634616 44.97062795 44.97958702 44.92278459, 49.21896745 50.34141739 48.01144443 50.7113197 53.70022118 53.09089083, 36.81422243 42.11125906 47.5120631 48.23488525 52.38174794 59.13939887, 43.95188126 40.72350856 39.07306079 39.36255459 38.9502675 37.87678828, 36.04681885 38.40384619 41.65701426 40.27273715 42.31758531 43.51008817, 33.85725812 39.69753213 48.37801187 43.89804409 44.25391719 44.2835234 ] while all_dmos = [30.96926305 55.12598144 68.47434569 63.19511855 44.82338636 39.05790258, 61.77650141 43.23640628 27.29835247 55.44590755 39.26934316 28.35209766, 29.65262045 41.59899252 55.96629109 71.88924176 34.53450825 17.19991484, 26.41774852 46.88645613 79.15402107 50.65216125 45.56089672 36.8136669, 64.98568555 49.49478196 29.56212976 68.14515278 49.32783463 39.70465263, 29.9807714 40.99972896 58.42660889 85.19789617 53.10270256 24.20572819, 31.1199481 51.57753506 64.18712587 61.9394302 52.56382728 39.07539394, 61.0901873 42.19819949 27.14502109 73.33284815 42.48086625 34.19254771, 26.21587958 40.64126874 52.39742685 70.05419936 42.42780915 26.91805556] I would get "lib\site-packages\scipy\optimize\minpack.py:834: OptimizeWarning: Covariance of the parameters could not be estimated category=OptimizeWarning)" several times during this step. And the results of PLCC and SROCC only (0.33308124176152726, 0.3385172479512102).

Can you tell me how to solve this problem and get the right result.

Thanks in advance, Hugh

— Reply to this email directly, view it on GitHub https://github.com/JoshuaEbenezer/ChipQA/issues/2, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADT5VDF66JNF2JZBNQ65EY3WXNEOJANCNFSM6AAAAAAU3JPVUE . You are receiving this because you are subscribed to this thread.Message ID: @.***>

JoshuaEbenezer commented 1 year ago

Hi Hugh

I've fixed cleaner_svr.py. Please check and see if it works now.

Regards Josh

IRON-HUGHY commented 1 year ago

Hi Hugh The optimization warning comes up when there's no good fit available for the mapping. This can happen when the predictions are very different from the scores, which I suppose is happening here. I've not tested ChipQA on the CSIQ database but I would expect the SRCC to be around 0.7 since BRISQUE and TLVQM get an SRCC of around that level (Table IV in https://arxiv.org/pdf/2206.14713.pdf). In line 67 of cleanersvr.py, I'm making the assumption that the first element of the '' separated list of the filenames in the dataframe corresponds to the content. If this is not the case, you have to define a 'content' column in the dataframe yourself. Also the code removes distortions marked as 'p', which in our original database corresponded to proprietary content. That might be one reason why performance is poor. I will update cleaner_svr.py to fix these things, but you may have to make more changes to the SVR code to make it run correctly for CSIQ, depending on how you created your score CSV. Regards, Josh … On Tue, Feb 14, 2023 at 3:16 AM IRON-HUGHY @.> wrote: Hi Joshua, I tried to use chipQA to train and test its performance on a CSIQ-VQA database, which is a 480p resolution database. 1 - I run chipqa_yuv.py get the feature file 2 - I run python cleaner_svr.py --score_file /path/to/score.csv --feature_folder feature_folder --train_and_test I got the all_pred = [33.22243675 49.20781647 50.0144947 41.65980157 41.57749611 40.87659356, 61.43243688 58.08024139 38.79844636 58.73480296 46.18201971 36.00328378, 36.96726173 42.92148637 53.04071207 63.78199282 53.71268326 31.00923555, 43.38495816 44.25350569 43.77634616 44.97062795 44.97958702 44.92278459, 49.21896745 50.34141739 48.01144443 50.7113197 53.70022118 53.09089083, 36.81422243 42.11125906 47.5120631 48.23488525 52.38174794 59.13939887, 43.95188126 40.72350856 39.07306079 39.36255459 38.9502675 37.87678828, 36.04681885 38.40384619 41.65701426 40.27273715 42.31758531 43.51008817, 33.85725812 39.69753213 48.37801187 43.89804409 44.25391719 44.2835234 ] while all_dmos = [30.96926305 55.12598144 68.47434569 63.19511855 44.82338636 39.05790258, 61.77650141 43.23640628 27.29835247 55.44590755 39.26934316 28.35209766, 29.65262045 41.59899252 55.96629109 71.88924176 34.53450825 17.19991484, 26.41774852 46.88645613 79.15402107 50.65216125 45.56089672 36.8136669, 64.98568555 49.49478196 29.56212976 68.14515278 49.32783463 39.70465263, 29.9807714 40.99972896 58.42660889 85.19789617 53.10270256 24.20572819, 31.1199481 51.57753506 64.18712587 61.9394302 52.56382728 39.07539394, 61.0901873 42.19819949 27.14502109 73.33284815 42.48086625 34.19254771, 26.21587958 40.64126874 52.39742685 70.05419936 42.42780915 26.91805556] I would get "lib\site-packages\scipy\optimize\minpack.py:834: OptimizeWarning: Covariance of the parameters could not be estimated category=OptimizeWarning)" several times during this step. And the results of PLCC and SROCC only (0.33308124176152726, 0.3385172479512102). Can you tell me how to solve this problem and get the right result. Thanks in advance, Hugh — Reply to this email directly, view it on GitHub <#2>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADT5VDF66JNF2JZBNQ65EY3WXNEOJANCNFSM6AAAAAAU3JPVUE . You are receiving this because you are subscribed to this thread.Message ID: @.>

Hi Josh, Thank you very much for your reply, but it doesn't appear to be the problem with the.csv file, because the video file in the CSIQ database is named very similar to the one in livestream, and the content of the video is before the first '_' separator.I adjusted the sigmoid entry in curve_fit () to make it less easy to overflow encountered in exp. def sigmoids(inx): if inx >= 0: # return 1.0 / (1 + np.exp(-inx)) else: return np.exp(inx) / (1 + np.exp(inx)) preds_fitted fitted with this adjustment, def sigmoid(x): x_ravel = x.ravel() # numpy length = len(x_ravel) y = [] for index in range(length): if x_ravel[index] >= 0: y.append(1.0 / (1 + np.exp(-x_ravel[index]))) else: y.append(np.exp(x_ravel[index]) / (np.exp(x_ravel[index]) + 1)) return np.array(y).reshape(x.shape)

and no warning appeared after the adjustment.But there does not seem to be a fitting operation. In the end, I tried to ignore these warnings, and the end result was (PLCC,SROCC )only (0.43037323037323044, 0.4201873570797587).It could be this database data is no good fit available for the mapping.

Regards, Hugh

JoshuaEbenezer commented 1 year ago

Hi Hugh

Ok. In any case, the fitting will not be necessary for NR algorithms because the SVR is mapping to the score space. Fitting is more relevant for FR algorithms which output some fidelity measure which needs to be mapped to scores.

Also, these results are usually reported as median results for 100 content-separated train-test splits. In addition, using a linear SVR kernel is known to sometimes give better performance than RBF kernels for these features.

If you've done all these things then I suppose this database is just a challenging one for ChipQA.

Regards, Josh

On Wed, Feb 15, 2023 at 2:12 AM IRON-HUGHY @.***> wrote:

Hi Hugh The optimization warning comes up when there's no good fit available for the mapping. This can happen when the predictions are very different from the scores, which I suppose is happening here. I've not tested ChipQA on the CSIQ database but I would expect the SRCC to be around 0.7 since BRISQUE and TLVQM get an SRCC of around that level (Table IV in https://arxiv.org/pdf/2206.14713.pdf). In line 67 of cleanersvr.py, I'm making the assumption that the first element of the '' separated list of the filenames in the dataframe corresponds to the content. If this is not the case, you have to define a 'content' column in the dataframe yourself. Also the code removes distortions marked as 'p', which in our original database corresponded to proprietary content. That might be one reason why performance is poor. I will update cleaner_svr.py to fix these things, but you may have to make more changes to the SVR code to make it run correctly for CSIQ, depending on how you created your score CSV. Regards, Josh … <#m_6900202025710888295m-6479947538803208170_> On Tue, Feb 14, 2023 at 3:16 AM IRON-HUGHY @.> wrote: Hi Joshua, I tried to use chipQA to train and test its performance on a CSIQ-VQA database, which is a 480p resolution database. 1 - I run chipqa_yuv.py get the feature file 2 - I run python cleaner_svr.py --score_file /path/to/score.csv --feature_folder feature_folder --train_and_test I got the all_pred = [33.22243675 49.20781647 50.0144947 41.65980157 41.57749611 40.87659356, 61.43243688 58.08024139 38.79844636 58.73480296 46.18201971 36.00328378, 36.96726173 42.92148637 53.04071207 63.78199282 53.71268326 31.00923555, 43.38495816 44.25350569 43.77634616 44.97062795 44.97958702 44.92278459, 49.21896745 50.34141739 48.01144443 50.7113197 53.70022118 53.09089083, 36.81422243 42.11125906 47.5120631 48.23488525 52.38174794 59.13939887, 43.95188126 40.72350856 39.07306079 39.36255459 38.9502675 37.87678828, 36.04681885 38.40384619 41.65701426 40.27273715 42.31758531 43.51008817, 33.85725812 39.69753213 48.37801187 43.89804409 44.25391719 44.2835234 ] while all_dmos = [30.96926305 55.12598144 68.47434569 63.19511855 44.82338636 39.05790258, 61.77650141 43.23640628 27.29835247 55.44590755 39.26934316 28.35209766, 29.65262045 41.59899252 55.96629109 71.88924176 34.53450825 17.19991484, 26.41774852 46.88645613 79.15402107 50.65216125 45.56089672 36.8136669, 64.98568555 49.49478196 29.56212976 68.14515278 49.32783463 39.70465263, 29.9807714 40.99972896 58.42660889 85.19789617 53.10270256 24.20572819, 31.1199481 51.57753506 64.18712587 61.9394302 52.56382728 39.07539394, 61.0901873 42.19819949 27.14502109 73.33284815 42.48086625 34.19254771, 26.21587958 40.64126874 52.39742685 70.05419936 42.42780915 26.91805556] I would get "lib\site-packages\scipy\optimize\minpack.py:834: OptimizeWarning: Covariance of the parameters could not be estimated category=OptimizeWarning)" several times during this step. And the results of PLCC and SROCC only (0.33308124176152726, 0.3385172479512102). Can you tell me how to solve this problem and get the right result. Thanks in advance, Hugh — Reply to this email directly, view it on GitHub <#2 https://github.com/JoshuaEbenezer/ChipQA/issues/2>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADT5VDF66JNF2JZBNQ65EY3WXNEOJANCNFSM6AAAAAAU3JPVUE https://github.com/notifications/unsubscribe-auth/ADT5VDF66JNF2JZBNQ65EY3WXNEOJANCNFSM6AAAAAAU3JPVUE . You are receiving this because you are subscribed to this thread.Message ID: @.>

Hi Josh, Thank you very much for your reply, but it doesn't appear to be the problem with the.csv file, because the video file in the CSIQ database is named very similar to the one in livestream, and the content of the video is before the first '_' separator.I adjusted the sigmoid entry in curve_fit () to make it less easy to overflow encountered in exp. def sigmoids(inx): if inx >= 0: # return 1.0 / (1 + np.exp(-inx)) else: return np.exp(inx) / (1 + np.exp(inx)) preds_fitted fitted with this adjustment, def sigmoid(x): x_ravel = x.ravel() # numpy length = len(x_ravel) y = [] for index in range(length): if x_ravel[index] >= 0: y.append(1.0 / (1 + np.exp(-x_ravel[index]))) else: y.append(np.exp(x_ravel[index]) / (np.exp(x_ravel[index]) + 1)) return np.array(y).reshape(x.shape)

and no warning appeared after the adjustment.But there does not seem to be a fitting operation. In the end, I tried to ignore these warnings, and the end result was (PLCC,SROCC )only (0.43037323037323044, 0.4201873570797587).It could be this database data is no good fit available for the mapping.

Regards, Hugh

— Reply to this email directly, view it on GitHub https://github.com/JoshuaEbenezer/ChipQA/issues/2#issuecomment-1430916820, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADT5VDCFU4IU3GO375BOBEDWXSFV3ANCNFSM6AAAAAAU3JPVUE . You are receiving this because you commented.Message ID: @.***>

IRON-HUGHY commented 1 year ago

Hi Josh

Sorry for the late reply. I have used a linear SVR kernel and tried 1000 times content-separated train-test splits. The result of PLCC and SROCC did improve, and now the results have become(0.522230860287501, 0.5335907335907336).

Regards, Hugh