search

SysCV / sam-hq

Segment Anything in High Quality [NeurIPS 2023]
https://arxiv.org/abs/2306.01567
Apache License 2.0
3.65k stars 219 forks source link

IoU Computation Problem #84

Open MrZilinXiao opened 11 months ago

MrZilinXiao commented 11 months ago

Thank you for your great work! I would like to discuss something about how HQ-SAM produces val_iou during training.

https://github.com/SysCV/sam-hq/blob/5fd027ec559806dec9f876b56938175a63dfc625/train/train.py#L601-L603

I don't think reducing IoU value across GPUs will give a faithful metric. Instead, the correct way should be to reduce pixel numbers of area_intersect, area_union, area_pred_label, and area_label.

The same happens when you try to average IoU on all predictions inside a batch.

https://github.com/SysCV/sam-hq/blob/5fd027ec559806dec9f876b56938175a63dfc625/train/train.py#L509-L511

ymq2017 commented 11 months ago

Hi, thanks for watching our work! Here we treat each image/object/mask equally instead of each pixel. For example, if we have two images with one big elephant mask and one small ant mask. If we sum the intersection and union, then divide, the image of the small ant will nearly be ignored.

MrZilinXiao commented 11 months ago

Sure! Your explanation is reasonable as it aligns well with how you implement IoU evaluation. I am just double-confirming that this is not common in other segmentation evaluation toolkits.

Thanks for the clarification!