Unseen classes in COCO

[Dopamine0717]

Hi, I have found that the ground truth of unseen classes in COCO have been used in trainning, I wonder if I'm misunderstood.

ids of unseen classes in COCO

output of model, as you can see, 'selected_id' contain '36' which is one of the unseen classes in COCO

while bipartite matching, the gt of label 36 has been used in computing the cost

[xiaohuihui52309]

+1

[Hzzone]

+1 I have not found whether the unseen classes have been excluded during training. I have downloaded the annotation file instances_train2017_seen_2_proposal.json, and found that the unseen classes exist in annotations. Please refer to https://github.com/alirezazareian/ovr-cnn/blob/master/ipynb/003.ipynb that has indeed excluded the unseen classes.

[huzhangcs]

I also found the same problem but no explanation for now

[yechenzhi]

https://github.com/yuhangzang/OV-DETR/issues/5 you can check this issue to see how the author got instances_train2017_seen_2_proposal.json file. I also visualized some images by instances_train2017_seen_2_proposal and coco_train_anno_all seperately, for example when image_id = 176179, in instances_train2017_all_2: in instances_train2017_seen_2_proposal:

[Solacex]

This question is simple, the issue raiser compares the label indexes before and after the label mapping, the mentioned list self.cat_ids_unseen stores the unseen label ids before label mapping, while the output class 36 is the id after mapping.

[huzhangcs]

Then what about the self.all_ids here https://github.com/yuhangzang/OV-DETR/blob/main/ovdetr/models/model.py#L290 self.all_ids is in range(0,64), which is exactly the same as seen_classes+unseen_class after mapping. No differences are observed when they process seen and unseen classes. They just randomly select the classes from self.all_ids, and randomly select embeddings from the pkl file based on the selected classes. The labels of unseen classes are leaked in this process as i can see.

[yechenzhi]

Then what about the self.all_ids here https://github.com/yuhangzang/OV-DETR/blob/main/ovdetr/models/model.py#L290 self.all_ids is in range(0,64), which is exactly the same as seen_classes+unseen_class after mapping. No differences are observed when they process seen and unseen classes. They just randomly select the classes from self.all_ids, and randomly select embeddings from the pkl file based on the selected classes. The labels of unseen classes are leaked in this process as i can see.

no, the annotations used in OV-DETR for unseen classes are not real annotations, but generated by a class agnostic detector and then classified by CLIP.

[huzhangcs]

I understand what you mean, but please check how they use their clip text embedding in https://github.com/yuhangzang/OV-DETR/blob/main/ovdetr/models/model.py#L294, self.zeroshot_w includes the groundtruth text embeddings for unseen classes. If randomly select id from self.all_ids, someone could use the groundtruth text embeddings directly in the training. Do you think it is still the open-voc problem?

[yechenzhi]

I understand what you mean, but please check how they use their clip text embedding in https://github.com/yuhangzang/OV-DETR/blob/main/ovdetr/models/model.py#L294, self.zeroshot_w includes the groundtruth text embeddings for unseen classes. If randomly select id from self.all_ids, someone could use the groundtruth text embeddings directly in the training. Do you think it is still the open-voc problem?

you are right, the training process of OV-DETR includes unseen classes, and it differs from previous Open-Voc settings, which may be an unfair setting.

[huzhangcs]

According to the paper, this should not be the case. But based on the code released, they just use the groundtruth text label for unseen classes directly.

[yechenzhi]

According to the paper, this should not be the case. But based on the code released, they just use the groundtruth text label for unseen classes directly.

you can check table 1 in https://arxiv.org/pdf/2303.13076v1.pdf , 'Require Novel Class', I guess there are different settings in open-voc problem.

[huzhangcs]

No, still not correct. If you investigate similar work carefully, for example, ViLD which also requires novel classes. The setting should be that you can pose some constraints to potential objects, which potentially include novel objects. However, in the released code of OV-DETR, they directly constrain the novel objects directly. These two cases are completely different.