search

Yikai-Wang / ICI-FSL

This repository contains the code for Instance Credibility Inference (ICI).
82 stars 22 forks source link

Question on experimental results #16

Closed instop1 closed 2 years ago

instop1 commented 2 years ago

Hi, thanks for your excellent work! I am a beginner of artificial intelligence. I am trying to run the experiments on the tieredImageNet dataset (1-shot, unlabel=15) and getting the results: 70.23 75.71 76.70 76.79 76.36 0.510 0.582 0.607 0.620 0.621 The first column represents the precision of the pre-training model at the beginning of the test, and the second to the fourth columns represent the precision obtained after each sample addition. What does the value of the last column represent? Which of these five values is the final result of the experiment? Thanks in advance.

Yikai-Wang commented 2 years ago

Hi there, Thanks for your attention to our work. (1) Our algorithm select 5 pseudo-labeled instance for each class until all the data are included in the training set. Ideally, for the setting of 15 unlabeled data, the training size of each class is expanded as 1 (support set in 1-shot), 1+5, 1+10, 1+15. However, in practice when the classifier provide class-imbalanced pseudo-labels, the process may be 1+5, 1+10, 1+13, 1+14, 1+15, which leads to more iterations of running. Hence we simply print results of iteration 1 -> 1 iteration 2 -> 1+5 iteration 3 -> 1+10 iteration 4 -> 1+15 iteration 5 -> termination where iteration 5 is added to contain the class-imbalanced pseudo-label case. (2) As we illustrated in the above, the final iteration usually contains some bad cases where the termination result will introduce extra noise and result in worse performance. In our experiments, we will select the different choice of ending time and report the corresponding results, where the selection is different with different experiment settings but the same with different algorithms to avoid over-fitting the test data. Besides, the number is chosen based on the validation set rather than the test set, hence you may find a different choice may lead to better performance in some experiments. Please refer the Settings part in our paper (CVPR version or TPAMI version, based on the code you are running) to find the precise settings. Hope these can help you.

instop1 commented 2 years ago

Got it! Thanks for your prompt reply!