search

akshitac8 / Generative_MLZSL

[TPAMI 2023] Generative Multi-Label Zero-Shot Learning
https://akshitac8.github.io/GAN_MLZSL/
GNU General Public License v3.0
51 stars 14 forks source link

Evaluation File #10

Closed aliman80 closed 2 years ago

aliman80 commented 2 years ago
  1. The dataset argument given in eval.py is not defined.
  2. The HYBRID FUSION SELF ATTENTION module is not defined in file you need to change it to CLF.
  3. When i run the code using given pre-trained weights the accuracies are very low
  4. Without pre trained wieghts accuracies are in range of mAp 20 for zsl
akshitac8 commented 2 years ago

Hello @aliman80 , Thank you for the valuable suggestions. Can you please share your NUS-WIDE training result log file?

aliman80 commented 2 years ago

Hi Thank you very much for your response. i have attached the recent most training results for your consideration please.

regards

Ali

From: Akshita Gupta @.> Sent: Monday, January 17, 2022 2:07 PM To: akshitac8/Generative_MLZSL @.> Cc: aliman80 @.>; Mention @.> Subject: Re: [akshitac8/Generative_MLZSL] Evaluation File (Issue #10)

Hello @aliman80https://github.com/aliman80 , Thank you for the valuable suggestions. Can you please share your NUS-WIDE training result log file?

— Reply to this email directly, view it on GitHubhttps://github.com/akshitac8/Generative_MLZSL/issues/10#issuecomment-1014345682, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ANUKA7BSYAIKYOWRRQMKE7LUWPS4RANCNFSM5MATAPIQ. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.Message ID: @.***>

aliman80 commented 2 years ago

Training_File_Results.pdf

akshitac8 commented 2 years ago

Hi @aliman80 I have attached my log file with the message. In the log file you shared the encoder loss is very high. Please check if all the inputs are passed correctly to the code and also the uploaded code was developed on Tesla v100 machine. So, if you are using a different machine make sure your environment is developed correctly and if you are still not able to get the results you might have to do certain parameter changes to achieve near by results. pami_reproduce_git_nus_wide.log

aliman80 commented 2 years ago

Hi, Thank you very much for your response. I have used the code directly from the repository without any change . Any last comments what inputs specifically should i check and if you have any reference how much those inputs be so that i can compare.

Thanks again

ali

From: Akshita Gupta @.> Sent: Monday, January 17, 2022 3:15 PM To: akshitac8/Generative_MLZSL @.> Cc: aliman80 @.>; Mention @.> Subject: Re: [akshitac8/Generative_MLZSL] Evaluation File (Issue #10)

Hi @aliman80https://github.com/aliman80 I have attached my log file with the message. In the log file you shared the encoder loss is very high. Please check if all the inputs are passed correctly to the code and also the uploaded code was developed on Tesla v100 machine. So, if you are using a different machine make sure your environment is developed correctly and if you are still not able to get the results you might have to do certain parameter changes to achieve near by results. pami_reproduce_git_nus_wide.loghttps://github.com/akshitac8/Generative_MLZSL/files/7881374/pami_reproduce_git_nus_wide.log

— Reply to this email directly, view it on GitHubhttps://github.com/akshitac8/Generative_MLZSL/issues/10#issuecomment-1014406423, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ANUKA7D6JNYI7MYR2ZC4J23UWP24PANCNFSM5MATAPIQ. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.Message ID: @.***>

aliman80 commented 2 years ago

Another thing to understand the data loading do you have json files of NUS-WIDE , also it takes 30 hours of code run to complete training loop. If you can upload small dataset which can run early that will be great help.

Thanks again

Ali

From: muhammad ali @.> Sent: Monday, January 17, 2022 3:55 PM To: akshitac8/Generative_MLZSL @.> Subject: Re: [akshitac8/Generative_MLZSL] Evaluation File (Issue #10)

Hi, Thank you very much for your response. I have used the code directly from the repository without any change . Any last comments what inputs specifically should i check and if you have any reference how much those inputs be so that i can compare.

Thanks again

ali

From: Akshita Gupta @.> Sent: Monday, January 17, 2022 3:15 PM To: akshitac8/Generative_MLZSL @.> Cc: aliman80 @.>; Mention @.> Subject: Re: [akshitac8/Generative_MLZSL] Evaluation File (Issue #10)

Hi @aliman80https://github.com/aliman80 I have attached my log file with the message. In the log file you shared the encoder loss is very high. Please check if all the inputs are passed correctly to the code and also the uploaded code was developed on Tesla v100 machine. So, if you are using a different machine make sure your environment is developed correctly and if you are still not able to get the results you might have to do certain parameter changes to achieve near by results. pami_reproduce_git_nus_wide.loghttps://github.com/akshitac8/Generative_MLZSL/files/7881374/pami_reproduce_git_nus_wide.log

— Reply to this email directly, view it on GitHubhttps://github.com/akshitac8/Generative_MLZSL/issues/10#issuecomment-1014406423, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ANUKA7D6JNYI7MYR2ZC4J23UWP24PANCNFSM5MATAPIQ. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.Message ID: @.***>

akshitac8 commented 2 years ago

Hi @aliman80 you can check the classifier batch size, learning rate. further you can also check the ratio of seen, unseen labels used for preparing the synthetic data. You can compose the json files of NUS-WIDE dataset using the uploaded features. You can manipulate the size of the dataset in the data loader.

aliman80 commented 2 years ago

Thank you very much i will do it .

Thanks again

From: Akshita Gupta @.> Sent: Monday, January 17, 2022 8:06 PM To: akshitac8/Generative_MLZSL @.> Cc: aliman80 @.>; Mention @.> Subject: Re: [akshitac8/Generative_MLZSL] Evaluation File (Issue #10)

Hi @aliman80https://github.com/aliman80 you can check the classifier batch size, learning rate. further you can also check the ratio of seen, unseen labels used for preparing the synthetic data. You can compose the json files of NUS-WIDE dataset using the uploaded features. You can manipulate the size of the dataset in the data loader.

— Reply to this email directly, view it on GitHubhttps://github.com/akshitac8/Generative_MLZSL/issues/10#issuecomment-1014694933, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ANUKA7G27RZGFSKU65E5TEDUWQ5CBANCNFSM5MATAPIQ. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.Message ID: @.***>

aliman80 commented 2 years ago

One last clarification is that in the case of NUS-WIDE all the features and tags are given so I don't need any preprocessing/extraction . We only need that if we use custom datasets? as mentioned right?

From: muhammad ali @.> Sent: Monday, January 17, 2022 8:09 PM To: akshitac8/Generative_MLZSL @.> Subject: Re: [akshitac8/Generative_MLZSL] Evaluation File (Issue #10)

Thank you very much i will do it .

Thanks again

From: Akshita Gupta @.> Sent: Monday, January 17, 2022 8:06 PM To: akshitac8/Generative_MLZSL @.> Cc: aliman80 @.>; Mention @.> Subject: Re: [akshitac8/Generative_MLZSL] Evaluation File (Issue #10)

Hi @aliman80https://github.com/aliman80 you can check the classifier batch size, learning rate. further you can also check the ratio of seen, unseen labels used for preparing the synthetic data. You can compose the json files of NUS-WIDE dataset using the uploaded features. You can manipulate the size of the dataset in the data loader.

— Reply to this email directly, view it on GitHubhttps://github.com/akshitac8/Generative_MLZSL/issues/10#issuecomment-1014694933, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ANUKA7G27RZGFSKU65E5TEDUWQ5CBANCNFSM5MATAPIQ. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.Message ID: @.***>

aliman80 commented 2 years ago

As per your suggestion i checked the input features shape,

    train_loc = load_dict_from_hdf5(os.path.join(src, 'nus_wide_vgg_features','nus_seen_train_vgg19.h5'))
    test_unseen_loc = load_dict_from_hdf5(os.path.join(src, 'nus_wide_vgg_features', 'nus_zsl_test_vgg19.h5'))
    test_seen_unseen_loc = load_dict_from_hdf5(os.path.join(src, 'nus_wide_vgg_features', 'nus_gzsl_test_vgg19.h5'))

Here the value of test_unseen_loc vs test_seen_unseen_loc features values are exactly similar in shape , In my understanding they need to be different . So my question is the extracted features which you have provided in google drive for zsl and gzsl need to be same? i think it should be different. If it makes sense to you and if somehow value of same feature sets is given twice then please update them.

test_unseen_loc. (107859, 4096) test_seen_unseen_loc['features'].shape test_seen_unseen_loc (107859, 4096)

From: muhammad ali @.> Sent: Monday, January 17, 2022 8:19 PM To: akshitac8/Generative_MLZSL @.> Subject: Re: [akshitac8/Generative_MLZSL] Evaluation File (Issue #10)

One last clarification is that in the case of NUS-WIDE all the features and tags are given so I don't need any preprocessing/extraction . We only need that if we use custom datasets? as mentioned right?

From: muhammad ali @.> Sent: Monday, January 17, 2022 8:09 PM To: akshitac8/Generative_MLZSL @.> Subject: Re: [akshitac8/Generative_MLZSL] Evaluation File (Issue #10)

Thank you very much i will do it .

Thanks again

From: Akshita Gupta @.> Sent: Monday, January 17, 2022 8:06 PM To: akshitac8/Generative_MLZSL @.> Cc: aliman80 @.>; Mention @.> Subject: Re: [akshitac8/Generative_MLZSL] Evaluation File (Issue #10)

Hi @aliman80https://github.com/aliman80 you can check the classifier batch size, learning rate. further you can also check the ratio of seen, unseen labels used for preparing the synthetic data. You can compose the json files of NUS-WIDE dataset using the uploaded features. You can manipulate the size of the dataset in the data loader.

— Reply to this email directly, view it on GitHubhttps://github.com/akshitac8/Generative_MLZSL/issues/10#issuecomment-1014694933, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ANUKA7G27RZGFSKU65E5TEDUWQ5CBANCNFSM5MATAPIQ. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.Message ID: @.***>

aliman80 commented 2 years ago

Hi, By using pre trained weights mAP of zsl in range of 3 and GZSL in range of 1 which is quite far away. Can you update the right weights in pre trained folder please.