search

jameelhassan / PromptAlign

[NeurIPS 2023] Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalization
https://jameelhassan.github.io/promptalign/
93 stars 10 forks source link

reproducing results on ImageNet-V dataset #10

Open yjh576 opened 3 months ago

yjh576 commented 3 months ago

Despite successfully reproducing results on other datasets, I am unable to replicate the results of AlignPrompt on ImageNet-V using your code. I observe that the accuracy on ImageNet-V is almost zero, and it seems that the model is unable to converge. I confirm that I have not altered the code and have implemented it correctly. Could you kindly provide suggestions or assistance? Your assistance would be greatly appreciated.

jameelhassan commented 3 months ago

Can you share the results that you get for Imagenet-V?

yjh576 commented 3 months ago

Here are some of my reported results, and the model seems to neither converge nor perform effectively.

Re-updating prompt initializations to current prompts. => Using native Torch AMP. Training in mixed precision. number of test samples: 10000 Running for 64 Augmented views Running for 1 TTA steps Test: [ 199/10000] Time 0.879 ( 1.067) Acc@1 0.00 ( 1.00) Acc@5 0.00 ( 2.50) Test: [ 399/10000] Time 0.891 ( 0.977) Acc@1 0.00 ( 0.75) Acc@5 0.00 ( 2.00)

Bostoncake commented 3 months ago

I have the same issue reproducing results for v2. Did you manage to come up with some solutions?

Bostoncake commented 3 months ago

@jameelhassan I wonder if you would provide some insights on reproducing results for the v2 dataset? Thanks a lot!

jameelhassan commented 3 months ago

Hi @yjh576 and @Bostoncake The issue is in the dataset class. Unfortunately, I do not have access to the computing resources till mid-July. Will check this and update the code soon.

Iris1946 commented 1 month ago

hi @jameelhassan I was wondering if the issue with the ImageNet-v2 dataset not being reproducible has been resolved. Could you please let me know when you might be planning to update the code? Thank you very much for your help.