Open Jonas-Meier opened 3 years ago
Hi @Jonas-Meier
Thanks for spotting it and sorry for being not very responsive at this moment. This might be an actual issue, I will look further into it. My guess is that a new training strategy needs to be found if this is actually the case.
And potentially I will update the code with newer few-shot object detection code built upon detectron such like frustratingly simple ICML-2020 or contrastive CVPR-2021.
I probably found an issue with the indexing of categories at coco evaluation. This might cause interchanged results for each category which would also affect AP(novel) and AP(base) since it is unknown which result belongs to which category #30 . The overall mAP is unaffected by this issue.
At inference, the test-class puts the detections inside the variable
all_boxes
where the classes are indexed by their position inimdb.classes
(which is Background-Class + Base-Classes + Novel-Classes, as defined in the constructor of coco.py).As coco.py does the evaluation, the following happens:
evaluate_detections
gets the detections viaall_boxes
from the test classevaluate_detections
calls_write_coco_results_file
_write_coco_results_file
iterates over its categories (self.classes) and uses the mappingcoco_cat_id = self._class_to_coco_cat_id[cls]
to obtain the category id from category name. It passes each category ID to_coco_results_one_category
which will return detections in a different format:_do_detection_eval
_do_detection_eval
creates an instance of COCOeval with_do_detection_eval
runs evaluate and accumulate on the cocoeval object and passes it to_print_detection_eval_metrics
self.params.catIds = sorted(cocoGt.getCatIds())
, where cocoGt is the COCO-object initialized with the validation annotation fileNow we have two problematic situations inside _print_detection_eval_metrics() method of coco.py:
cls_ind, cls in enumerate(self.classes)
, but as stated above, the metrics for a class are stored at the index of that class as in the validation annotation file. This causes the category names for per-category results to be interchangedcocoeval.summarize
. However, thesummarize
method uses thecategoryId
argument to directly access the precision and recall of that class, but those indices are wrong (as described above for class-wise AP)To solve the stated problems, I would suggest the following changes (for
_print_detection_eval_metrics
, line 245-259)self._base_classes
andself._novel_classes
are the lists of base and novel class names which I used to create self._classes in the constructor.Some final thoughts on the issue:
categoryId
insidecocoeval.summarize
is a bit confusing, since they treat them as indicesself.classes
(of coco.py) and the categories as in the COCO objectself.classes
(instead of having to read in original coco categories for the correct order) and just have to do a simple index shift to obtain correct results for each class.