How to get IoU in validation metrics?

[vakker]

Hi,

I'm running an instance segmentation model (queryinst), and the validation logs contain the following:

{"mode": "val", "epoch": 1, "iter": 100, "lr": 0.0, 
"bbox_mAP": 0.979, "bbox_mAP_50": 0.983, "bbox_mAP_75": 0.983, "bbox_mAP_s": -1.0, 
"bbox_mAP_m": -1.0, "bbox_mAP_l": 0.979, "bbox_mAP_copypaste": "0.979 0.983 0.983 -1.000 -1.000 0.979", 
"segm_mAP": 0.921, "segm_mAP_50": 0.983, "segm_mAP_75": 0.983, "segm_mAP_s": -1.0, 
"segm_mAP_m": -1.0, "segm_mAP_l": 0.921, "segm_mAP_copypaste": "0.921 0.983 0.983 -1.000 -1.000 0.921"}

How can I include additional metrics, like segm/bbox IoU in this? Thanks

Edit: I've just noticed on larger validation sets that there are 2 separate entries for val in the log file, e.g. after epoch 8 I get this:

{"mode": "train", "epoch": 8, "iter": 7150, "lr": 5e-05, "memory": 9352, "data_time": 0.00686, "stage0_loss_cls": 0.0, "stage0_pos_acc": 100.0, "stage0_loss_bbox": 0.08268, "stage0_loss_iou": 0.0569, "stage0_loss_mask": 0.03742, "stage1_loss_cls": 0.0, "stage1_pos_acc": 100.0, "stage1_loss_bbox": 0.03378, "stage1_loss_iou": 0.02396, "stage1_loss_mask": 0.0348, "stage2_loss_cls": 0.0, "stage2_pos_acc": 100.0, "stage2_loss_bbox": 0.02977, "stage2_loss_iou": 0.02085, "stage2_loss_mask": 0.03426, "stage3_loss_cls": 0.0, "stage3_pos_acc": 100.0, "stage3_loss_bbox": 0.02862, "stage3_loss_iou": 0.02, "stage3_loss_mask": 0.03339, "stage4_loss_cls": 0.0, "stage4_pos_acc": 100.0, "stage4_loss_bbox": 0.02888, "stage4_loss_iou": 0.02012, "stage4_loss_mask": 0.03425, "stage5_loss_cls": 0.0, "stage5_pos_acc": 100.0, "stage5_loss_bbox": 0.02893, "stage5_loss_iou": 0.02018, "stage5_loss_mask": 0.03425, "loss": 0.60305, "grad_norm": 30.55146, "time": 0.44809}
{"mode": "val", "epoch": 8, "iter": 4772, "lr": 5e-05, "bbox_mAP": 0.989, "bbox_mAP_50": 0.99, "bbox_mAP_75": 0.99, "bbox_mAP_s": -1.0, "bbox_mAP_m": -1.0, "bbox_mAP_l": 0.989, "bbox_mAP_copypaste": "0.989 0.990 0.990 -1.000 -1.000 0.989", "segm_mAP": 0.99, "segm_mAP_50": 0.99, "segm_mAP_75": 0.99, "segm_mAP_s": -1.0, "segm_mAP_m": -1.0, "segm_mAP_l": 0.99, "segm_mAP_copypaste": "0.990 0.990 0.990 -1.000 -1.000 0.990"}
{"mode": "val", "epoch": 8, "iter": 2386, "lr": 5e-05, "memory": 9352, "data_time": 0.00707, "stage0_loss_cls": 0.0, "stage0_pos_acc": 100.0, "stage0_loss_bbox": 0.08284, "stage0_loss_iou": 0.05611, "stage0_loss_mask": 0.0345, "stage1_loss_cls": 0.0, "stage1_pos_acc": 100.0, "stage1_loss_bbox": 0.03609, "stage1_loss_iou": 0.02452, "stage1_loss_mask": 0.0358, "stage2_loss_cls": 0.0, "stage2_pos_acc": 100.0, "stage2_loss_bbox": 0.03275, "stage2_loss_iou": 0.02257, "stage2_loss_mask": 0.03543, "stage3_loss_cls": 0.0, "stage3_pos_acc": 100.0, "stage3_loss_bbox": 0.03332, "stage3_loss_iou": 0.0228, "stage3_loss_mask": 0.03543, "stage4_loss_cls": 0.0, "stage4_pos_acc": 100.0, "stage4_loss_bbox": 0.03324, "stage4_loss_iou": 0.02263, "stage4_loss_mask": 0.03591, "stage5_loss_cls": 0.0, "stage5_pos_acc": 100.0, "stage5_loss_bbox": 0.03384, "stage5_loss_iou": 0.02314, "stage5_loss_mask": 0.03572, "loss": 0.63663, "time": 0.18482}

My val set has 4772 elements, so why is the second val entry there in the middle of the validation iterations?

[RangiLyu]

COCO api does not support the IOU metric, it only supports bbox and instance segmentation mAP.

My val set has 4772 elements, so why is the second val entry there in the middle of the validation iterations? One of the 'val' is to evaluate, another 'val' is to post the losses on val dataset.

[vakker]

But the IOU calculation already exists in MMdet, as it's used for the mAP calculations (e.g. see recall.py), also there's IOU calculation in the training loss (e.g. stage0_loss_iou). So, I'm not sure how the COCO api is relevant here.

[RangiLyu]

Loss and evaluation metrics are two different concepts.

[vakker]

That's understood, but IOU is a metric commonly used for bbox and segm problems, and it would be trivial to add regardless of the COCO api, and in fact there's already an IOU calculation in the COCO evaluation, since it's required for the mAP calculations, right? So, would that be possible to report the bbox and segm (where applicable) IOU in the validation metrics?