How to reproduce the results of Baseline-FPN?

[Ze-Yang]

I would like to reproduce the results of Baseline-FPN. Is there any hyperparameter need to adjust? Or just using the same one. Also I notice that your Baseline-FPN results are somewhat different from Frustratingly Simple Few-Shot Object Detection while under the same few-shot training samples. Say 43.8 (yours) vs 39.0 (Frustratingly Simple Few-Shot Object Detection) on split2 10shot setting. Is it due to implementation? Thanks.

[jiaxi-wu]

Cfg files on Baseline-FPN is the same as this MPSR. You may need to modify the detection head for class-agnostic regression if working on the original maskrcnn-benchmark repo. Refer to our paper (section3.1) for more details. Hope this will help you.

[Ze-Yang]

Noted with thanks. How about the second question regarding the same data samples but different results? Thanks.

[jiaxi-wu]

I recommand to ask the authors of Frustratingly Simple Few-Shot Object Detection: why their performance of split-2 10-shot is so bad. You can compare these results with their previous work FSRW. In split-2, their results exceed FSRW at all the shot num except 10-shot. I'm curious about this, too. Thanks.

[Ze-Yang]

Okay, thanks for the advice.

[Ze-Yang]

Hi, following your instruction, I set the regressor as class agnostic and train it on VOC_split1_base. After that, I remove the last layer of the classifier and finetune with randomly initialized one on the FSRW 5 shot samples. The results are shown in the below pic. It still exists a gap with yours. Note that I use the same training strategy with your config file. Is there anything else need to pay attention to? Thanks.

[jiaxi-wu]

It seems the perfomance on base classes are also low. I guess something may be wrong during base training or fine-tuning. Can you show more info like:

1. mAP on test set of VOC_split1_base before fine-tuning.
2. Code repo you use.
3. Loss during fine-tuning.

[Ze-Yang]

FYI,

1. 82.16 mAP on VOC_split1_base
2. detectron2
3. please refer to the attached files below. Note that I rename the tensorboard output file into *.txt format since github portal only accepts limited file formats. Just using it with conventional command tensorboard --logdir xxx.

log.txt tensorboard.txt

[jiaxi-wu]

There are some different settings:

1. 8-gpu with 0.02 lr, 2000 steps. 4x schedule compared with our cfg. This might lead to overfitting.
2. MIN_SIZE_TRAIN: (480, 512, 544, 576, 608, 640, 672, 704, 736, 768, 800). I don't know the side effect of this scale aug.

I recommend a shorter training schedule temporarily.

[Ze-Yang]

Following your instruction, I get the results with base mAP 70.89 and novel mAP 44.77 for split1 10shot. Log file can be accessed here. Could you please provide the code to reproduce the Baseline-FPN? Generally there might be some different details across codebase, making it difficult to reproduce the results.

[jiaxi-wu]

Hi~ @Ze-Yang I will try to modify this repo for the Baseline-FPN exps, which may take ~1 week.