problem concerning the closed-set metrics

Jun-CEN / Open-world-3D-semantic-segmentation

[ECCV 2022] Open-world Semantic Segmentation for LIDAR Point Clouds

69 stars 10 forks source link

problem concerning the closed-set metrics #21

Closed Daniellli closed 1 year ago

Daniellli commented 1 year ago

thank you for your such novel contribution.

Why the evaluation code of $mIoU{old}$, is calculated by including IoU of novel classes (such as other-vehicle for semantic kitti) which are all equal to zero. This is inconsistent with the definition of $mIoU{old}$?

Do I understand correctly? Or am i missing something? please correct me, thank you~

Jun-CEN commented 1 year ago

Hi,

The code does show the mIoU of all classes including the unknown classes. However, we report the result of mIoU_old without unknown classes in the paper. As shown in Table 5, the mIoU_old is around 74.4 which clearly doesn't include those unknown classes with 0.