Are there utility methods for calculating AP metric for results from the model?

[Dayan-Zhanchi]

❓ Questions and Help

So let's say that I have trained a model on keypoints or instance segmentation with datasets that are on coco-format, and used that trained model to get some outputs (Inference) on an unseen data (test data). Now I want to compute the AP with OKS or IOU of that output. Are there any utility functions that can help with this or do we have to add it ourselves? I should say that all my datasets are in coco-format.

[wuyuanyi135]

see https://github.com/facebookresearch/detectron2/issues/320

[ppwwyyxx]

The colab tutorial (https://colab.research.google.com/drive/16jcaJoc6bCFAQ96jDe2HwtXj7BMD_-m5#scrollTo=kblA1IyFvWbT) is now updated to include inference & evaluation.

[Dayan-Zhanchi]

Thank you! So I tried running it on my coco-format dataset, following the colab tutorial for inference and evaluation, but it seems that when the number of keypoints is not 17, the evaluation will get a dimension error. As can be seen in the following error message:

``` --------------------------------------------------------------------------- ValueError Traceback (most recent call last) in () 4 evaluator = COCOEvaluator(NAME_OF_DATASET_VAL, cfg_keypoint, False, output_dir="./output/") 5 val_loader = build_detection_test_loader(cfg_keypoint, NAME_OF_DATASET_VAL) ----> 6 inference_on_dataset(trainer_keypoint.model, val_loader, evaluator) 7 predictor = DefaultPredictor(cfg_keypoint) /content/Detectron2-project/detectron2/evaluation/evaluator.py in inference_on_dataset(model, data_loader, evaluator) 151 ) 152 --> 153 results = evaluator.evaluate() 154 # An evaluator may return None when not in main process. 155 # Replace it by an empty dict instead to make it easier for downstream code to handle /content/Detectron2-project/detectron2/evaluation/coco_evaluation.py in evaluate(self) 136 self._eval_box_proposals() 137 if "instances" in self._predictions[0]: --> 138 self._eval_predictions(set(self._tasks)) 139 # Copy so the caller can do whatever with results 140 return copy.deepcopy(self._results) /content/Detectron2-project/detectron2/evaluation/coco_evaluation.py in _eval_predictions(self, tasks) 173 self._coco_api, self._coco_results, task, kpt_oks_sigmas=self._kpt_oks_sigmas 174 ) --> 175 if len(self._coco_results) > 0 176 else None # cocoapi does not handle empty results very well 177 ) /content/Detectron2-project/detectron2/evaluation/coco_evaluation.py in _evaluate_predictions_on_coco(coco_gt, coco_results, iou_type, kpt_oks_sigmas) 459 if kpt_oks_sigmas: 460 coco_eval.params.kpt_oks_sigmas = np.array(kpt_oks_sigmas) --> 461 coco_eval.evaluate() 462 coco_eval.accumulate() 463 coco_eval.summarize() /usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py in evaluate(self) 148 computeIoU = self.computeOks 149 self.ious = {(imgId, catId): computeIoU(imgId, catId) \ --> 150 for imgId in p.imgIds 151 for catId in catIds} 152 /usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py in (.0) 149 self.ious = {(imgId, catId): computeIoU(imgId, catId) \ 150 for imgId in p.imgIds --> 151 for catId in catIds} 152 153 evaluateImg = self.evaluateImg /usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py in computeOks(self, imgId, catId) 228 dx = np.max((z, x0-xd),axis=0)+np.max((z, xd-x1),axis=0) 229 dy = np.max((z, y0-yd),axis=0)+np.max((z, yd-y1),axis=0) --> 230 e = (dx**2 + dy**2) / vars / (gt['area']+np.spacing(1)) / 2 231 if k1 > 0: 232 e=e[vg > 0] ValueError: operands could not be broadcast together with shapes (294,) (17,) ```

Is there some way to customize this? I'm guessing that the cocoevaluator only has support for persons keypoints, which is 17 keypoints, but I have 294 keypoints in total over all my cloth categories (13 different categories with varying number of keypoints for each category, all summing up to in total 294 keypoints).

[ppwwyyxx]

I'm not sure if it's related to the OKS settings in cocoapi, but detectron2's config allows you to set OKS for cocoapi.

[Dayan-Zhanchi]

I haven't looked thoroughly at it, but ye I think you are right. So it's then up to me to define the individual sigma values for the OKS. It seems that for the coco evaluation they use: sigmas = np.array([.26, .25, .25, .35, .35, .79, .79, .72, .72, .62,.62, 1.07, 1.07, .87, .87, .89, .89])/10.0, but I obviously will need other values.

EDIT: So I think I found the formula for OKS:


[Dayan-Zhanchi]

For people working with deepfashion2, the calculated sigma values can be found in the authors repository in the cocoeval.py file: https://github.com/switchablenorms/DeepFashion2/blob/4ba6aae3bbcbf9013fe43f3ce9cc5bfb033639ae/deepfashion2_api/PythonAPI/pycocotools/cocoeval.py#L206

I'll just put it here in case people working with deepfashion2 needs it in the future. NOTE that this is only valid if you are working with deepfashion2 dataset, otherwise you need to calculate the sigma values by yourself, following section 1.3 in http://cocodataset.org/#keypoints-eval.

sigmas = np.array([0.012 , 0.0158, 0.0169, 0.0165, 0.0169, 0.0158, 0.0298, 0.0329, 0.0321, 0.0323, 0.034 , 0.0388, 0.0452, 0.0574, 0.0492, 0.0352, 0.0492, 0.0574, 0.0452, 0.0388, 0.034 , 0.0323, 0.0321, 0.0329, 0.0298, 0.0194, 0.017 , 0.0185, 0.0193, 0.0185, 0.017 , 0.0286, 0.0471, 0.0547, 0.0526, 0.043 , 0.0392, 0.0513, 0.0566, 0.0509, 0.0564, 0.0542, 0.0604, 0.0599, 0.052 , 0.0599, 0.0604, 0.0542, 0.0564, 0.0509, 0.0566, 0.0513, 0.0392, 0.043 , 0.0526, 0.0547, 0.0471, 0.0286, 0.0074, 0.0085, 0.0165, 0.0248, 0.0165, 0.0085, 0.0156, 0.0231, 0.0296, 0.0137, 0.0195, 0.025 , 0.0347, 0.038 , 0.0257, 0.037 , 0.0257, 0.038 , 0.0347, 0.025 , 0.0195, 0.0137, 0.0296, 0.0231, 0.0156, 0.0248, 0.0469, 0.0632, 0.037 , 0.0469, 0.0632, 0.0137, 0.0153, 0.0243, 0.0377, 0.0243, 0.0153, 0.0203, 0.0366, 0.0467, 0.0433, 0.0393, 0.0329, 0.0418, 0.0477, 0.0399, 0.0331, 0.042 , 0.0492, 0.0436, 0.0478, 0.0436, 0.0492, 0.042 , 0.0331, 0.0399, 0.0477, 0.0418, 0.0329, 0.0393, 0.0433, 0.0467, 0.0366, 0.0203, 0.0377, 0.0645, 0.0573, 0.0478, 0.0645, 0.0573, 0.0352, 0.0158, 0.021 , 0.0214, 0.021 , 0.0158, 0.0196, 0.05 , 0.0489, 0.0404, 0.0401, 0.0404, 0.0489, 0.05 , 0.0196, 0.0276, 0.0548, 0.0283, 0.0204, 0.0283, 0.0548, 0.0369, 0.0726, 0.0677, 0.064 , 0.0251, 0.064 , 0.0677, 0.0726, 0.0369, 0.0308, 0.0216, 0.0308, 0.0506, 0.0494, 0.0463, 0.0477, 0.0463, 0.0494, 0.0506, 0.0275, 0.0202, 0.0275, 0.0651, 0.0451, 0.035 , 0.028 , 0.0392, 0.0362, 0.0392, 0.028 , 0.035 , 0.0451, 0.0651, 0.0253, 0.0195, 0.0253, 0.0513, 0.0543, 0.0415, 0.0543, 0.0513, 0.0153, 0.023 , 0.0167, 0.0145, 0.0167, 0.023 , 0.0332, 0.0391, 0.0391, 0.0396, 0.044 , 0.0452, 0.0498, 0.0514, 0.0585, 0.0655, 0.0635, 0.0602, 0.0635, 0.0655, 0.0585, 0.0514, 0.0498, 0.0452, 0.044 , 0.0396, 0.0391, 0.0391, 0.0332, 0.0121, 0.0134, 0.0158, 0.0162, 0.0158, 0.0134, 0.0246, 0.0406, 0.047 , 0.0404, 0.0463, 0.0466, 0.0435, 0.0499, 0.0455, 0.044 , 0.0411, 0.049 , 0.0576, 0.0685, 0.0618, 0.0483, 0.0618, 0.0685, 0.0576, 0.049 , 0.0411, 0.044 , 0.0486, 0.0499, 0.0435, 0.0466, 0.0463, 0.0404, 0.047 , 0.0406, 0.0246, 0.0116, 0.0167, 0.016 , 0.018 , 0.016 , 0.0167, 0.0196, 0.0385, 0.0421, 0.0497, 0.0562, 0.0528, 0.0428, 0.0528, 0.0562, 0.0497, 0.0421, 0.0385, 0.0196, 0.0244, 0.0297, 0.0244, 0.0208, 0.0244, 0.0297, 0.0173, 0.0616, 0.0659, 0.0712, 0.0707, 0.0685, 0.0339, 0.0685, 0.0707, 0.0712, 0.0659, 0.0616, 0.0173])