Unexpected Results On MiniImageNet With Pretrain ResNet

[KZF-kzf]

hello,Hi, I'm having some problems replicating your results. 2024-07-17 12:23:23,497 - meta_test.py[line:154] - INFO: max |k: 0.26 |mix acc: 74.56+0.65% |gap: 11.82 2024-07-17 12:23:23,497 - meta_test.py[line:156] - INFO: ACC:|proto acc: 62.74+0.80% |gen acc: 70.96+0.69% My experiment is set up as follows: 1.I used the ResNet-MinilmageNet.pth file provided in your cloud disk as the weight of compute_center.py 2.Execute train_seman_l1_cen.py to get the epoch_best-pth file, and then execute the meta-test, with the result shown above. 3.The other Settings remain the default or the same as you github gave Now there seems to be some gap between the results I got and those in the paper. Is there anything wrong with my Settings?

[KZF-kzf]

train.txt this is my train log

[DruryXu]

Sorry for the late reply. Could you please check whether the dataset used is correct?

[KZF-kzf]

Thank you very much for your reply. Here is the data set I used, and I think that should be fine. Please let me know if you need more details. Thank you, and for your excellent work.

[myl666666]

我看到作者的GUTHUB上的work-dir 是 resnet,你的好像是CNN

[DruryXu]

Thank you very much for your reply. Here is the data set I used, and I think that should be fine. Please let me know if you need more details. Thank you, and for your excellent work.

The way you use MiniImageNet in code is correct, but there are many different versions of the miniimagenet dataset, please confirm which version of miniimagenet you are using. If possible, please use the miniimagenet which is approximately 3GB in size.

[DruryXu]

我看到作者的GUTHUB上的work-dir 是 resnet,你的好像是CNN

They are the same thing with different names, we modified this name for the sake of consistency.

[KZF-kzf]

Thank you very much for your reply. Here is the data set I used, and I think that should be fine. Please let me know if you need more details. Thank you, and for your excellent work.

The way you use MiniImageNet in code is correct, but there are many different versions of the miniimagenet dataset, please confirm which version of miniimagenet you are using. If possible, please use the miniimagenet which is approximately 3GB in size.

Thanks for your reply.Can you give a download link of the MiniImageNet used in your code?

[MeycL]

Have you found all the correct datasets? (miniimagenet, tieredimagenet, cifar-fs, fc100) I have looked for publicly available data online to train, but I can't reproduce the results in the paper (all are lower). If possible, could you please provide the links to the correct datasets? Thank you very much.

[DruryXu]

Have you found all the correct datasets? (miniimagenet, tieredimagenet, cifar-fs, fc100) I have looked for publicly available data online to train, but I can't reproduce the results in the paper (all are lower). If possible, could you please provide the links to the correct datasets? Thank you very much.

Could you please share your reproduction results？

[MeycL]

Have you found all the correct datasets? (miniimagenet, tieredimagenet, cifar-fs, fc100) I have looked for publicly available data online to train, but I can't reproduce the results in the paper (all are lower). If possible, could you please provide the links to the correct datasets? Thank you very much.

Could you please share your reproduction results？

Thank you very much for the prompt response.

In the miniimagenet experiments, the results for 5-way 1-shot are 71.87+0.68, while for another 5-way 1-shot it is 77.55+0.56. This clearly differs significantly from the results presented in the paper, and similar issues are observed in other datasets.

My experimental environment is: Ubuntu 20.04, PyTorch 1.11.0, CUDA: 11.3. All code and pre-trained weights are the latest versions, and the experimental steps follow the readme. Since this project does not provide the dataset used, I referred to FewTURE for my dataset. During my attempts, I noticed that the dataset seems to affect the results. While checking other issues, I found many questions related to the dataset. Could the authors provide a Google Drive link or another clearly accessible link to all the data (miniimagenet, tieredimagenet, cifar-fs, fc100) used? I believe this would help everyone's learning and the dissemination of this work.

Look forward to your reply.

[DruryXu]

Have you found all the correct datasets? (miniimagenet, tieredimagenet, cifar-fs, fc100) I have looked for publicly available data online to train, but I can't reproduce the results in the paper (all are lower). If possible, could you please provide the links to the correct datasets? Thank you very much.

Could you please share your reproduction results？

Thank you very much for the prompt response.

In the miniimagenet experiments, the results for 5-way 1-shot are 71.87+0.68, while for another 5-way 1-shot it is 77.55+0.56. This clearly differs significantly from the results presented in the paper, and similar issues are observed in other datasets.

My experimental environment is: Ubuntu 20.04, PyTorch 1.11.0, CUDA: 11.3. All code and pre-trained weights are the latest versions, and the experimental steps follow the readme. Since this project does not provide the dataset used, I referred to FewTURE for my dataset. During my attempts, I noticed that the dataset seems to affect the results. While checking other issues, I found many questions related to the dataset. Could the authors provide a Google Drive link or another clearly accessible link to all the data (miniimagenet, tieredimagenet, cifar-fs, fc100) used? I believe this would help everyone's learning and the dissemination of this work.

Look forward to your reply.

This is a weird result, is this result comes from resnet or swin tiny? Thank you for reminding us, we will check whether the problem comes from bugs during our code reorganization. We will also consider the dataset suggestion, thank you.

[MeycL]

Have you found all the correct datasets? (miniimagenet, tieredimagenet, cifar-fs, fc100) I have looked for publicly available data online to train, but I can't reproduce the results in the paper (all are lower). If possible, could you please provide the links to the correct datasets? Thank you very much.

Could you please share your reproduction results？

Thank you very much for the prompt response. In the miniimagenet experiments, the results for 5-way 1-shot are 71.87+0.68, while for another 5-way 1-shot it is 77.55+0.56. This clearly differs significantly from the results presented in the paper, and similar issues are observed in other datasets. My experimental environment is: Ubuntu 20.04, PyTorch 1.11.0, CUDA: 11.3. All code and pre-trained weights are the latest versions, and the experimental steps follow the readme. Since this project does not provide the dataset used, I referred to FewTURE for my dataset. During my attempts, I noticed that the dataset seems to affect the results. While checking other issues, I found many questions related to the dataset. Could the authors provide a Google Drive link or another clearly accessible link to all the data (miniimagenet, tieredimagenet, cifar-fs, fc100) used? I believe this would help everyone's learning and the dissemination of this work. Look forward to your reply.

This is a weird result, is this result comes from resnet or swin tiny? Thank you for reminding us, we will check whether the problem comes from bugs during our code reorganization. We will also consider the dataset suggestion, thank you.

Swin tiny.

[DruryXu]

Have you found all the correct datasets? (miniimagenet, tieredimagenet, cifar-fs, fc100) I have looked for publicly available data online to train, but I can't reproduce the results in the paper (all are lower). If possible, could you please provide the links to the correct datasets? Thank you very much.

Could you please share your reproduction results？

Thank you very much for the prompt response. In the miniimagenet experiments, the results for 5-way 1-shot are 71.87+0.68, while for another 5-way 1-shot it is 77.55+0.56. This clearly differs significantly from the results presented in the paper, and similar issues are observed in other datasets. My experimental environment is: Ubuntu 20.04, PyTorch 1.11.0, CUDA: 11.3. All code and pre-trained weights are the latest versions, and the experimental steps follow the readme. Since this project does not provide the dataset used, I referred to FewTURE for my dataset. During my attempts, I noticed that the dataset seems to affect the results. While checking other issues, I found many questions related to the dataset. Could the authors provide a Google Drive link or another clearly accessible link to all the data (miniimagenet, tieredimagenet, cifar-fs, fc100) used? I believe this would help everyone's learning and the dissemination of this work. Look forward to your reply.

This is a weird result, is this result comes from resnet or swin tiny? Thank you for reminding us, we will check whether the problem comes from bugs during our code reorganization. We will also consider the dataset suggestion, thank you.

Swin tiny.

We have uploaded datasets

[DruryXu]

Thank you very much for your reply. Here is the data set I used, and I think that should be fine. Please let me know if you need more details. Thank you, and for your excellent work.

The way you use MiniImageNet in code is correct, but there are many different versions of the miniimagenet dataset, please confirm which version of miniimagenet you are using. If possible, please use the miniimagenet which is approximately 3GB in size.

Thanks for your reply.Can you give a download link of the MiniImageNet used in your code?

We have uploaded datasets

[KitABC12]

Hello, have you reproduced the author's results using the MiniImageNet dataset now?

[MeycL]

Hello, have you reproduced the author's results using the MiniImageNet dataset now?

I regret to say that, despite the author providing the corresponding datasets, I still achieved suboptimal results across all datasets. Some results were close to or even worse than FewTURE, far from the outstanding performance described in the paper. Therefore, I believe I may not be able to follow this work, and I wonder if others have encountered similar issues. Of course, I do not rule out the possibility that I made an error at some step.

[DruryXu]

Hello, have you reproduced the author's results using the MiniImageNet dataset now?

I regret to say that, despite the author providing the corresponding datasets, I still achieved suboptimal results across all datasets. Some results were close to or even worse than FewTURE, far from the outstanding performance described in the paper. Therefore, I believe I may not be able to follow this work, and I wonder if others have encountered similar issues. Of course, I do not rule out the possibility that I made an error at some step.

Thank you for your feedback. If possible, I would like to obtain your latest reproduction results, especially those that are worse than FewTURE.

[MeycL]

Hello, have you reproduced the author's results using the MiniImageNet dataset now?

I regret to say that, despite the author providing the corresponding datasets, I still achieved suboptimal results across all datasets. Some results were close to or even worse than FewTURE, far from the outstanding performance described in the paper. Therefore, I believe I may not be able to follow this work, and I wonder if others have encountered similar issues. Of course, I do not rule out the possibility that I made an error at some step.

Thank you for your feedback. If possible, I would like to obtain your latest reproduction results, especially those that are worse than FewTURE. swin_FC100_clip_gpt_mean_5_train.txt swin_FC100_clip_gpt_mean_1_train.txt

Here is my experiment log on the FC100 (Environment: PyTorch 1.12.1 CUDA 11.6). Other data also show similar results, but I lost the logs. I can retrain and provide them once the holiday is over.

[DruryXu]

Hello, have you reproduced the author's results using the MiniImageNet dataset now?

I regret to say that, despite the author providing the corresponding datasets, I still achieved suboptimal results across all datasets. Some results were close to or even worse than FewTURE, far from the outstanding performance described in the paper. Therefore, I believe I may not be able to follow this work, and I wonder if others have encountered similar issues. Of course, I do not rule out the possibility that I made an error at some step.

Thank you for your feedback. If possible, I would like to obtain your latest reproduction results, especially those that are worse than FewTURE. swin_FC100_clip_gpt_mean_5_train.txt swin_FC100_clip_gpt_mean_1_train.txt

Here is my experiment log on the FC100 (Environment: PyTorch 1.12.1 CUDA 11.6). Other data also show similar results, but I lost the logs. I can retrain and provide them once the holiday is over.

If you remember the approximate accuracy of the model on the test set, that would also be helpful, especially for those datasets with significant performance differences.

[MeycL]

If you remember the approximate accuracy of the model on the test set, that would also be helpful, especially for those datasets with significant performance differences.

Thank you for your reply. Perhaps I can provide the specific data after the holiday.

[MeycL]

Hello, have you reproduced the author's results using the MiniImageNet dataset now?

I regret to say that, despite the author providing the corresponding datasets, I still achieved suboptimal results across all datasets. Some results were close to or even worse than FewTURE, far from the outstanding performance described in the paper. Therefore, I believe I may not be able to follow this work, and I wonder if others have encountered similar issues. Of course, I do not rule out the possibility that I made an error at some step.

Thank you for your feedback. If possible, I would like to obtain your latest reproduction results, especially those that are worse than FewTURE. swin_FC100_clip_gpt_mean_5_train.txt swin_FC100_clip_gpt_mean_1_train.txt

Here is my experiment log on the FC100 (Environment: PyTorch 1.12.1 CUDA 11.6). Other data also show similar results, but I lost the logs. I can retrain and provide them once the holiday is over.

If you remember the approximate accuracy of the model on the test set, that would also be helpful, especially for those datasets with significant performance differences.

Dear author,

Hello! I am still encountering inconsistencies with the results of your paper during my reproduction efforts. May I kindly ask if you could provide the logs from the experimental process and the pre-trained model? I believe this would save some time. Thank you!

[DruryXu]

Hello, have you reproduced the author's results using the MiniImageNet dataset now?

I regret to say that, despite the author providing the corresponding datasets, I still achieved suboptimal results across all datasets. Some results were close to or even worse than FewTURE, far from the outstanding performance described in the paper. Therefore, I believe I may not be able to follow this work, and I wonder if others have encountered similar issues. Of course, I do not rule out the possibility that I made an error at some step.

Thank you for your feedback. If possible, I would like to obtain your latest reproduction results, especially those that are worse than FewTURE. swin_FC100_clip_gpt_mean_5_train.txt swin_FC100_clip_gpt_mean_1_train.txt

Here is my experiment log on the FC100 (Environment: PyTorch 1.12.1 CUDA 11.6). Other data also show similar results, but I lost the logs. I can retrain and provide them once the holiday is over.

If you remember the approximate accuracy of the model on the test set, that would also be helpful, especially for those datasets with significant performance differences.

Dear author,

Hello! I am still encountering inconsistencies with the results of your paper during my reproduction efforts. May I kindly ask if you could provide the logs from the experimental process and the pre-trained model? I believe this would save some time. Thank you!

Hi, could you please provide your reproduction results first? In your previous reply, you provided two training logs, but that's not what we wanted. The results in the training logs are on the validation set, not the test set. Could you provide the reproduction results on the test sets of each dataset?

[caoziyang1997]

你好，你好，我在复制您的结果时遇到了一些问题。 2024-07-17 12:23:23,497 - meta_test.py[line:154] - INFO: max |k: 0.26 |mix acc: 74.56+0.65% |gap: 11.82 2024-07-17 12:23:23,497 - meta_test.py[line:156] - INFO: ACC:|proto acc: 62.74+0.80% |gen acc: 70.96+0.69% 我的实验设置如下： 1.使用你们云盘提供的ResNet-MinilmageNet.pth文件作为compute_center.py的权重 2.执行train_seman_l1_cen.py得到epoch_best-pth文件，然后执行meta-test，结果如上图。 3.其他设置保持默认或者和github给的一样 现在我得到的结果和论文里的结果好像有点差距，是我的设置有问题吗？

Could you please share your dataset with me? Thank you very much!

[qttis]

Thank you very much for your reply. Here is the data set I used, and I think that should be fine. Please let me know if you need more details. Thank you, and for your excellent work.

The way you use MiniImageNet in code is correct, but there are many different versions of the miniimagenet dataset, please confirm which version of miniimagenet you are using. If possible, please use the miniimagenet which is approximately 3GB in size.

Dear author, could you please share a link to the tiered-imagenet-kwon dataset? Or could you provide an tiered-imagenet approximate number of GB?Thank you for your excellent work.