search

Yikai-Wang / ICI-FSL

This repository contains the code for Instance Credibility Inference (ICI).
82 stars 22 forks source link

different experiment results with CUB pre-trained model #8

Closed MichalisLazarou closed 3 years ago

MichalisLazarou commented 3 years ago

Dear Yikai, Congrats on your work and many thanks for making the code available. I am trying to run the experiments on the CUB dataset using the pre-trained model but I am getting around -15% less accuracy while I get similar accuracy for the rest of the datasets. I was wondering if there is any mistake on the CUB pth.tar file and if you have the original one that you got you results.

Regard, Michalis

Yikai-Wang commented 3 years ago

The reason is probably that we used the cropped images, as stated in our paper. In order to to make a fair comparison with [34], we crop all images of CUB with the bounding box provided by [51]. You may check the Datasets part of Experiments Section in our paper or [34] for more details.

MichalisLazarou commented 3 years ago

Hey Yikai,

Many thanks on your quick reply. I was wondering is there somewhere the code to apply this bounding box so that I use the same crop on the CUB images?

Yikai-Wang commented 3 years ago

We use the official bounding box, which you can find in the original dataset

MichalisLazarou commented 3 years ago

Perfect I did that and I got similar results

MichalisLazarou commented 3 years ago

One issue I still have is with mini-imagenet, I tried to repicate the results but I got around 3% lower in 1-shot and 1.5% lower in 5-shot over 600 iterations. I was wondering if there was anything I am missing using miniImagenet that leads to such difference

Yikai-Wang commented 3 years ago

Have you tried setting different random seeds? We found about 1% fluctuation in the accuracy. If the gap still exists, the reason may be the dataset. Actually recently we found that there are several miniImageNet exists and different data will result in different results. You may check a miniImageNet from here to see if the performance changes.

MichalisLazarou commented 3 years ago

Indeed that might be the reason of the dataset. Do you use the pkl files for tiered, cifar and miniImagenet as given in the metaoptnet repository?

MichalisLazarou commented 3 years ago

I managed to replicate the results on miniImagenet, but using the tieredImagenet from MetaOptNet seems to be completely off - did you train on a different tieredImagenet than the one provided from MetaOptNet?

Yikai-Wang commented 3 years ago

I am afraid that it is a little difficult to determine the source of the data now. The data we used was downloaded by others who has left the laboratory. No matter what kind of data set is used, one thing that can be determined is that the effectiveness of ICI is consistent. The absolute value of performance is related to the performance of the baseline classifier.

MichalisLazarou commented 3 years ago

Agreed on this Yikai. I would like to note that this is not to challenge your algorithm as i really like your method, it is mostly because I would like to use it on my work and therefore I am trying to get some objective results.

Yikai-Wang commented 3 years ago

Thank you for your interest in our work. Perhaps the best way to solve your problem is to upload the dataset we use. But we don’t have the permission to distribute it again, so I’m sorry we can’t do this. Since you have reproduced the results of part of these datasets, if you want to further verify the effectiveness of ICI, maybe you can try to use some other datasets beyond the datasets used in our paper. You may do some visualization or check out proof of the effectiveness of ICI under certain conditions. Hope it can help you.

MichalisLazarou commented 3 years ago

No worries, I totally understand. I think also in the semi-supervised setting the results are very close to your results so I assume that perhaps the biggest error was due to the seeding. Just one question i have on the semi-supervised regarding the CUB, when you use a big split lets say 80 images per class, how do you do that since most of the classes in the CUB dataset have around only 60 images? do you use some sort of augmentation or do you sample again the same images?

Yikai-Wang commented 3 years ago

As we stated in the Setting part of Section 4.1, when the number of images available are not enough to reach our setting, we will simply use all the residual images except images for support and query set. Hence the number is smaller than our setting, say, 50 or 80 unlabeled data. Since the problem only exist in the CUD dataset, we did not introduce other complicated operations for convenience. Practically, it is always worth to try some augmentation strategies to expand the unlabeled set.

Yikai-Wang commented 3 years ago

Hi there, the tieredImageNet we used should come from here. You may check it to reproduce the results.