Readme file wrong data?

[Nan-S]

Did you paste data wrongly for CIFAR100, ResNet18, 5-way 5-shot (Fig.4 (b)) in the readme file? Looks it's the same as the MiniImageNet one: miniImageNet, ResNet18, 5-way 5-shot (Fig.4 (d))? Can I know how you calculate the accuracy for each session? I run the resnet18-ft-cnn.sh for cifar100 without -resume-from './params/CIFAR100/resnet18/baseline/baseline_resnet18_cifar_64.10.params' and I get first session(base session) best accuracy at 70.23% which is higher than the Fig.4 (b) Ft-CNN first session around 64%. Not sure why this big gap happened.

[xyutao]

@Nan-S You are right!!! The table "CIFAR100, ResNet18, 5-way 5-shot" is a wrong duplicate with the miniImageNet one. We have fixed the mistake and reported the correct one.

The reported results in the tables are the initial ones in the paper implemented with our earlier code, where the CIFAR100 training script did not use the 'random crop' data augmentation or other training tricks, e.g., a larger weight_decay.

In the latest versions, we have been further boosting the performances by adding more training tricks to make it more practicable into real use. So, you may start from a higher base model to get better results if necessary.

[Nan-S]

Thanks for update! I am still confused about how to calculate the accuracy in the new sessions, for example, in resnet18-ft-cnn.sh for cifar100, I get:

session : 1
...
[Epoch 0] lr=0.1000 train=0.0000 | val(base)=0.6830 val(novel)=0.0000 overall=0.6305 | loss=0.99411666 anc l
oss=0.00000000 pdl loss:0.00000000 oce loss: 0.00000000 time: 1.20614529
best select : base: 0.683000 novel: 0.000000
[Epoch 1] lr=0.1000 train=0.0000 | val(base)=0.6135 val(novel)=0.0000 overall=0.5663 | loss=0.97155619 anc l
oss=0.00000000 pdl loss:0.00000000 oce loss: 0.00000000 time: 1.43696070
best select : base: 0.613500 novel: 0.000000
...
[Epoch 20] lr=0.1000 train=0.0000 | val(base)=0.0838 val(novel)=0.1200 overall=0.0866 | loss=0.57024127 anc
loss=0.00000000 pdl loss:0.00000000 oce loss: 0.00000000 time: 1.08268785
...
[Epoch 68] lr=0.0100 train=1.0000 | val(base)=0.0028 val(novel)=0.6780 overall=0.0548 | loss=0.08897255 anc
loss=0.00000000 pdl loss:0.00000000 oce loss: 0.00000000 time: 1.08337998
best select : base: 0.083833 novel: 0.120000
[Epoch 69] lr=0.0100 train=1.0000 | val(base)=0.0027 val(novel)=0.6800 overall=0.0548 | loss=0.08757918 anc
loss=0.00000000 pdl loss:0.00000000 oce loss: 0.00000000 time: 0.96224475
best select : base: 0.083833 novel: 0.120000

the best select is Epoch 20, which val(base)=0.0838 val(novel)=0.1200 overall=0.0866, the overall accuracy looks far from the reported accuracy of FT-CNN session 1 is 13.09%, I didn't find a clear defination about how to calculate this number, why not use the largest overall accuracy?

[xyutao]

@Nan-S In the main paper, we reported the overall accuracy as in class-incremental learning setting, which measures the recognition performance on the joint set of the test data of base + encountered novel classes, e.g., 65 classes after the first incremental learning session. However, we've found such kind of measurement leads to a highest overall accuracy when the number of new classes is much smaller than that of the old classes. That's to say, we might get the maximum overall accuracy without learning any new classe at the first incremental learning session.

[Epoch 0] lr=0.1000 train=0.0000 | val(base)=0.6830 val(novel)=0.0000 overall=0.6305

This will cause the model fail to learn new class and behaves worse after longer sequence of incremental learning.

Therefore, we've developed the acquisition-memory curves (in Fig.7 of supplementary material) to justify the selection of models, where a F score is computed to balance the accuracies of old and new classes. In this way, we can get a better recognition performance for longer sequence of learning.

[Nan-S]

Where can I see the paper with the supplementary materials? the 2 versions of the paper at https://arxiv.org/abs/2004.10956 both don't have the supplementary materials.

[xyutao]

@Nan-S Access the CVPR website http://conferences.computer.org/cvpr, and download the supplementary material.

username: cvpr20 passwd: conf20//

[Nan-S]

oh, got it! Thanks!

[xyutao]

A tip: avoid greedly over-parameterizing the model. Otherwise, a highly optimized model for current session can lead to bad generalization performances at the subsequent sessions.

[wuyujack]

Hi @Nan-S and @xyutao! May I know about your GPU, MXNet, and CUDA version for running the baseline experiments of session 0? I am using an RTX 2080Ti.

[Nan-S]

@wuyujack I am using MXNet 1.7 and CUDA 10.1 on the RTX 2080Ti

[Nan-S]

@xyutao Can I know how you run the base session for cub200? I can only get 43.71% with the cub200 scripts resnet18-ft-cnn.sh without --resume-from, did you use any pretrain model?

[xyutao]

@Nan-S Yes. The CUB200 base model is initialized with ImageNet-pretrained model from gluon's model zoo.

[Nan-S]

Ok, I see. Thanks for your help! I will proceed to close this issue.

[xyutao]

@Nan-S @wuyujack I've uploaded the pretrained base models and imagenet pretrained models to a netdisk. Please refer to params/README.md for how to access these models.

[Nan-S]

@xyutao Can you also release your split reference file for miniimagenet? the slplit file I found all have train,val,test which is different from the split method used in fscil

[xyutao]

@Nan-S Hi, a test file containing test images has been uploaded. When evaluating the model of session_t, select the images of all the encounter classes from session_1 to session_t. May refer to https://github.com/xyutao/fscil/blob/63fc5ad3e2eb8be0d5525cd53462aa4044a70756/dataloader/dataloader.py#L152 for details.