Evaluation

[chrysts]

Hi, thanks for sharing the training sets. I want to know how 5-way 5-shot in the evaluation stage is constructed. Do you sample 5-way and 5-shot from the training set (all classes and samples from the specific session)?

Please correct me if I'm wrong. So let's say we are at session 2 (adding 5 classes) and there are 60+5 classes. 5-way are sampled from 65 classes and 5-shot are sampled from the training set (60 x initialsamples + 5x5). -> This is an episode. Then for each episode, all queries from the rest of the samples (not included in the training set) are evaluated?

[xyutao]

Hi~ You may have some misunderstandings about the FSCIL setting. It is NOT a few-shot learning setting. So we do not sample episodes for training & evaluation.

FSCIL is an incremental learning setting. That's to say, at session 2, the 60-class base training set is unseen, and we only have 5x5 new class training samples onsite (not sample a 5x5 episode.) After incrementally learn the model using the 25 samples, we evaluate the model to perform 1-v.s.-65 classification. We accumulate all the test samples from the 60-class base test set and the 5-class new test set as the joint test set. Then we feed every test image to the model and calculate the top-1 accuracy over 65 classes, just as ordinary image classification model does.

For example, CIFAR100 has 100 test images per class. At session 2, the test set for 60+5 classes has 100 x 65 samples, and the model is required to recognize all these samples.

[chrysts]

Thanks for your reply. OK fine, because there is a sentence "adopt 5-way 5-shot setting". I am a bit confused because of it. The 5-way 5-shot protocol only applies for additional sessions and not for the base set.

Another thing is that I saw at each session, there are already selected samples. But this sentence also makes me confused "As the 5-shot training samples are randomly picked". Do you have the other sessions that you haven't shared yet? I really want to put this setup to my work. So, I just want to be clear :).

[xyutao]

You may notice several key points in the context between Section 3 and 3.1 that may help you understand the setting: (1) a unified classification layer. (2) only D^t (t>1) is available at the t-th training session. (3) tested to recognize all encountered classes. (4) for D^t, C-classes and K-training samples.

'5-shot training samples are randomly picked' explains how the FSCIL benchmark datasets are constructed before the experiments, not how to sample episodes in the experiments. When constructing the training set D^t (t>1) before the experiments, we randomly select 5 images per class and then record them in the data list file for reproduction.

[chrysts]

yes yes, thanks for your explanation. It helps me to understand the protocol for evaluation. Now, I am clear about the experiments.

I am closing the issue.

[xyutao]

@chrysts May refer to a new README.md at folder "data/index_list"

[chrysts]

Thanks, I really appreciate it.