search

zhangdoudou123 / SemFew

[CVPR2024] Simple Semantic-Aided Few-Shot Learning
18 stars 4 forks source link

Unexpected Results on CIFAR-FS and FC100 Datasets with Pre-trained Swin_Vit Model #3

Closed jinqimu closed 2 weeks ago

jinqimu commented 1 month ago

Hello, I conducted experiments on the CIFAR-FS and FC100 datasets, using the FewTURE pre-trained _SwinVit 800-epoch weights, and other configurations refer to the code provided in the README file. The results I obtained are as follows:

CIFAR-FS: 2024-07-13 21:25:15,088 - meta_test_vit.py[line:137] - INFO: max |k: 0.08 |mix acc: 78.80+0.72% |gap: 22.00 2024-07-13 21:25:15,088 - meta_test_vit.py[line:139] - INFO: ACC:|proto acc: 56.80+0.80% |gen acc: 76.31+0.80%

FC100: 2024-07-13 20:09:20,510 - meta_test_vit.py[line:137] - INFO: max |k: 0.11 |mix acc: 48.64+0.72% |gap: 8.68 2024-07-13 20:09:20,510 - meta_test_vit.py[line:139] - INFO: ACC:|proto acc: 39.95+0.67% |gen acc: 41.09+0.73%

There is a discrepancy with the results provided in the paper. Are there any details I am not aware of?

DruryXu commented 1 month ago

The term 'proto acc' refers to the classification accuracy calculated using the embeddings output directly from the backbone through a cosine classifier. However, the value of your log on this term is significantly low. Please carefully check whether the model checkpoint or dataset used is correct.

jinqimu commented 1 month ago

The term 'proto acc' refers to the classification accuracy calculated using the embeddings output directly from the backbone through a cosine classifier. However, the value of your log on this term is significantly low. Please carefully check whether the model checkpoint or dataset used is correct.

After downloading the FewTURE Swin checkpoint, the loaded dictionary contains the following keys: ['student', 'teacher', 'optimizer', 'epoch', 'args', 'ibot_loss', 'fp16_scaler']. I chose to load 'teacher', but in your code, 'param' was loaded. Could this be the source of the problem?

DruryXu commented 1 month ago

Thank you for reminding us of this issue. We will upload the checkpoints we use to facilitate your reproduction. Please stay tuned.

DruryXu commented 1 month ago

We have uploaded new model weights. Please try to reproduce on this set of weights. We apologize for any inconvenience caused and welcome further questions if there are any issues.

jinqimu commented 1 month ago

We have uploaded new model weights. Please try to reproduce on this set of weights. We apologize for any inconvenience caused and welcome further questions if there are any issues.

Great job! I have now obtained the results in the paper. Why were the weights I used before not working?

DruryXu commented 1 month ago

We further trained the Swin-Tiny backbone using the FewTURE method. Our method focuses on the contribution of semantics in reconstructing visual prototypes, which works on features extracted by different visual backbones.

MeycL commented 1 week ago

Have you found all the correct datasets? (miniimagenet, tieredimagenet, cifar-fs, fc100) I have looked for publicly available data online to train, but I can't reproduce the results in the paper (all are lower). If possible, could you please provide the links to the correct datasets? Thank you very much.

DruryXu commented 1 week ago

Have you found all the correct datasets? (miniimagenet, tieredimagenet, cifar-fs, fc100) I have looked for publicly available data online to train, but I can't reproduce the results in the paper (all are lower). If possible, could you please provide the links to the correct datasets? Thank you very much.

Could you please share your reproduction results?