search

YoungXIAO13 / FewShotDetection

(ECCV 2020) PyTorch implementation of paper "Few-Shot Object Detection and Viewpoint Estimation for Objects in the Wild"
http://imagine.enpc.fr/~xiaoy/FSDetView/
MIT License
212 stars 33 forks source link

Report data leakage that causes unfair few-shot setting #27

Closed Ze-Yang closed 3 years ago

Ze-Yang commented 3 years ago

As I mentioned in yanxp/MetaR-CNN#36, the released code of Meta R-CNN exists a bug in shot sampling.

Take coco 10shot for example, you will first construct a meta-data set encomprising 30 (3x shots) (prn_image, prn_mask) pairs for each class. The image indexes of these sampled images are saved in a file named annotations/instances_shot2014.json.

After that, roidb needs to be contructed to provide query samples for finetuning purpose. According to the definition of few-shot setting, you can only access to the N-shot (N instances per class) data no matter whether you perform finetuning or not. Therefore, the roidb dataset should contain the same instances as in meta-data set, otherwise it will exceed the designated number of shots. However, as I find in your code, you does not save the anno_index of the selected instances in meta-data set. Instead, you again randomly sample the shot instances from the images list indicated in annotations/instances_shot2014.json. In this case, I am concern about how you guarantee that the newly sampled instances are exactly the same as the ones in meta-data set.

Hope that you can check about this issue and clarify my concern. Thanks a lot.

MonsterLWR commented 3 years ago

does the accuracy drops after merging?