test metric on custom dataset

[zhszysrh]

I need to test metric on my dataset, i follow the instruction ( https://github.com/mlzxy/devit/issues/4) to build CLASS_PROTOTYPES, my dataset has 69 classes. I use the default BG_PROTOTYPES: "weights/initial/background/background_prototypes.vitl14.pth", i get the error, how can i fix it

[mlzxy]

Hi @zhszysrh, you could torch.load the existing class prototypes and comparing the structure and tensor shape with yours. For example, the ones in this folder https://drive.google.com/drive/u/1/folders/1NF4JSQny7RsF99kNIqOAKggRkeqLD1tX.

Another thing, the evaluator (https://github.com/mlzxy/devit/blob/main/tools/train_net.py#L95) shall share the same class space with your prototypes. It seems your exception is raised from the evaluator. I recommend you step through the evaluator and see why the number of classes is mismatched.

I also use the class names to filter evaluation results (https://github.com/mlzxy/devit/blob/main/detectron2/evaluation/coco_evaluation.py#L370C22-L370C22) to make sure class indexes are not mismatch. So maybe check that part as well, like see are the values of self.seen_cnames and self.unseen_cnames expected.

[zhszysrh]

i actually don't need BG_PROTOTYPES，but i set BG_PROTOTYPES="" get error too.

[mlzxy]

It appears the error comes from the evaluator, which has no connection with BG_PROTOTYPES. I suggest you set BG_PROTOTYPES to original value and look into the corresponding evaluation code around line 183 of coco_evaluation.py, as the error indicated.

[zhszysrh]

I use the default BG_PROTOTYPES: "weights/initial/background/background_prototypes.vitl14.pth", is it the original value you suggest? The shape of "weights/initial/background/background_prototypes.vitl14.pth" is torch.Size([530, 1024]), my CLASS_PROTOTYPES is a dict which has key 'prototypes', and the value shape is torch.Size([69, 1024]). As the error indicated, the prediction has class = 284, i think 284 is strange. i think len(prediction class) should be len(CLASS_PROTOTYPES)(69) + len(BG_PROTOTYPES)(530) , is that wrong?

[mlzxy]

The number of predicted classes shall be 69. I suggest you set a breakpoint at https://github.com/mlzxy/devit/blob/main/detectron2/modeling/meta_arch/devit.py#L490C1-L490C1 and inspect the all_cids. It is configured through https://github.com/mlzxy/devit/blob/main/detectron2/modeling/meta_arch/devit.py#L791.

The reason could be I hardcode some internal dataset meta data in https://github.com/mlzxy/devit/blob/main/lib/categories.py. This greatly helps me to handle COCO/LVIS but may fail at customized datasets. You may need to add your dataset meta data (e.g., class names) to the categories.py.