search

chengchunhsu / WSIS_BBTP

Implementation of NeurIPS 2019 paper "Weakly Supervised Instance Segmentation using the Bounding Box Tightness Prior"
MIT License
97 stars 14 forks source link

Issues with IoU Computation in The Eval Code #11

Open voidrank opened 4 years ago

voidrank commented 4 years ago

https://github.com/chengchunhsu/WSIS_BBTP/blob/03cb87b9d9fe587346387b460e43200e73e4f86e/matlab/EvalVOCInstSeg.m#L75

The code computes the intersection over union (IoU) between the ground-truth mask and the predicted mask.

There are two severe issues in the above line.

  1. Both TempProposal and TempGTInst are matrices (2D tensors), sum(TempProposal & TempGTInst) and sum( TempProposal | TempGTInst) are vectors (1D tensors), which represent the intersection and union of each column of segmentation images. In Matlab, if we use the operation / to connect 2 vectors (let's say vector a divided by vector b), it will output a scalar r, such that |a - b * r| is minimal. However, r is used as IoU by mistake. I didn't see any relation between r and IoU and I check several r values during evaluation, none of them is equals to the correct IoU values.

  2. When debugging this line, I found that Intersection eliminates those ignored pixels but Union does not.

chengchunhsu commented 4 years ago

Hi @voidrank,

Thanks for pointing it out. We have been aware of this problem and will release an errata soon.

For now, you can fix this by changing the line
Overlap(j,i) = sum(TempProposal & TempGTInst) / sum(TempProposal | TempGTInst); as Overlap(j,i) = sum(sum(TempProposal & TempGTInst)) / sum(sum(TempProposal | TempGTInst));

Cheng-Chun

voidrank commented 4 years ago

Hi @chengchunhsu

Thanks for the confirmation.

Will be great if you can also pay attention to the second issue I mentioned. There may be another bug of how ignored pixels are handled in computing IoU.

voidrank commented 4 years ago

To be more specific, the ignored pixels means those edge pixels marked as 255th category, which should be ignored in computing IoU. However, I found that the evaluation code doesn't consider ignored pixels in computing intersection but takes ignored pixels into account when computing union.

chengchunhsu commented 4 years ago

Sure. I will get back to you after I detect the problem. Thanks for the effort.

Cheng-Chun