How to calculate the novel class AP in base training

[Fly-dream12]

Thanks for your code, i'm confused with the results reported in Table 4 in the paper. As is suggested, when doing the base training using Baseline-FPN, the base AP50 on 5 shot is 67.9 and the novel AP50 is 49.6. I wonder how it is evaluated. I run this command python tools/train_net.py --num-gpus 2 --config-file configs/PASCAL_VOC/base_training/R101_FPN_base_training_split1.yaml

The base AP50 is 77.458. I change the DATASETS.TEST in config file to (voc_2007_test_all1) to see the performance on novel class. However, it is 0.

Please explain it, thanks.

[Chauncy-Cai]

Because during base training step, we only train base class. That's to say, no instance of novel classes is trained. If my memory is right, during the base training stage, the classifier's shape weight is [61,1024], which means the model can't predict any novel class.

[Fly-dream12]

Thanks. So how is the Baseline-FPN results in Table 4 calculated ? (on both base and novel classes) @Chauncy-Cai

[bsun0802]

Thanks. So how is the Baseline-FPN results in Table 4 calculated ? (on both base and novel classes) @Chauncy-Cai

@Fly-dream12 Hi, we forget the reference in Table 4. We added the reference to the paper in an update to the arxiv. Please refer to the new version.

The first two lines, Baseline-FPN and MPSR, are directly coming from the paper MPSR (https://arxiv.org/abs/2007.09384). The 3rd and 4th lines are ours results. The method are different. Please refer to that paper for details.

[bsun0802]

Thanks. So how is the Baseline-FPN results in Table 4 calculated ? (on both base and novel classes) @Chauncy-Cai

Clearly the base detector model can not predict novel instances. The Base AP and Novel AP are for the novel fine-tuned model. The point is, after fine-tune, which method "forget" less about base.