search

JonasSchult / Mask3D

Mask3D predicts accurate 3D semantic instances achieving state-of-the-art on ScanNet, ScanNet200, S3DIS and STPLS3D.
MIT License
528 stars 105 forks source link

Potential bug with label_offset during evaluation of Scannet200 #34

Closed xuanlinli17 closed 1 year ago

xuanlinli17 commented 1 year ago

During the data collation process here, the code subtracts label_offset from the remapped semantic id (i.e. the semantic id obtained after _remap_from_zero in here) and clamps the resulting semantic id to be at least zero. During the evaluation process here, the code adds label_offset back to the semantic id.

However, this poses an issue for Scannet200 since its ignored remapped semantic ids are [0,2] with label_offset=2 (see here). In Scannet200, 0 in the remapped semantic id corresponds to the wall, 1 the chair, and 2 the floor. When performing evaluation, the 1 of the chair will become 0 after subtracting label_offset and then clamping to be at least zero. It will then become 2 during evaluation after adding label_offset back to 0, which causes the floor class to correspond to the chair class.

JonasSchult commented 1 year ago

Hi Xuanlin!

Thanks for your interest in our work!

Admittedly, the label_offset strategy is quite complicated and for sure, can be improved. You are right. You need to take special provisions in order to make it work for ScanNet200. Here, we take care of the floor class problem.

Best, Jonas