Wrong-looking calculation of the IoU metric

[cnmicha]

Dear authors,

Thank you very much for your work. I appreciate the idea of the paper.

I am currently trying to understand how the IoU metric calculation works in detail.

The code seems to be in this file: https://github.com/ywyue/RoomFormer/blob/33ad087f50f15fce04719828db8d2a56fd4b630f/scenecad_eval/Evaluator.py

In line 362, you iterate over all rooms in the current scene (gt_room_map_list). They are ordered by the number of points in the mask of the room (lines 347 and 344), in descending order.

Inside the for loop, you initialize the variable best_iou with value 0 (line 364) and overwrite it with the iou between prediction and target, in case a threshold is exceeded (line 376). So far, everything makes sense to me.

However, in line 441, you return the current value of best_iou, which is the iou between prediction and target of the room with the smallest number of points in the scene. You are not averaging over all rooms' IoU values in the scene. However, this is what I expected before returning a value in the function.

Currently, it seems like all rooms in each scene except for the room with the smallest number of points are being ignored in the IoU metric calculation.

Is this intended behavior or a bug? @ywyue

Thank you in advance.

[ywyue]

Hi @cnmicha , thanks for the detailed question! This is intended behavior (a hack implementation of the evaluator for SceneCAD, as indicated in the header of the file). The reason is that the evaluation datasets of SceneCAD only contain a single room for each floorplan, thus it will not lead to the issue you mentioned. Let me know if you have any further concerns!