The evaluation method used for your model is different from SurroundOcc.

[getterupper]

In your current config, you are using SegMetric from mmdet3d for evaluation, which calls function seg_eval. However, you have set ignore_index to 0, which causes the function to ignore numerous voxel predicted as noise during evaluation. As a result, the mIoU obtained from this method is actually higher than the mIoU obtained from evaluation method by SurroundOcc - in fact, according to the metric you are using for evaluation, the predicted mIoU for SurroundOcc would be $35$, instead of the current value you are comparing with, which is $20.30$. If you want to align your evaluation results with SurroundOcc, you should set ignore_index to 255 and exclude the noise category when calculating mIoU. According to this evaluation method, the mIoU predicted by your model should be around $13$. 😅

As shown in the visualization results below, the model's predictions are actually not as good as SurroundOcc. This is because a large number of voxels are predicted as noise, and this aspect somewhat positively impacts the mIoU:)

[DanielMing123]

Please check my loading.py code. class 0 dose not mean noise but refer to empty class. I first change all class 0 noise instances to 255, then set all empty classes as 0. During evaluation I exclude all voxels that has been classified as empty and noise voxel which has 255 semantic class is not being considered at all.

[getterupper]

Please check my loading.py code. class 0 dose not mean noise but refer to empty class. I first change all class 0 noise instances to 255, then set all empty classes as 0. During evaluation I exclude all voxels that has been classified as empty and noise voxel which has 255 semantic class is not being considered at all.

In your approach, you exclude voxels with the label of 0 before sum the voxel num, while SurroundOcc does not exclude this portion. Instead, SurroundOcc calculates the IoU for each class and then excludes this category when computing the overall mIoU. The former method leads to a lower number of FP than the actual situation, thereby increasing the calculated mIoU.

Under this evaluation method, your model indeed has an mIoU of $26.38$, but SurroundOcc would have an mIoU of $31.05$:

Meanwhile, if your model's mIoU is calculated using the SurroundOcc method, it would be $15.84$:

So it's:

	seg_eval	SurroundOcc's method
InverseMatrixVT3D	$26.38$	$15.84$
SurroundOcc	$31.05$	$20.30$

😂.

[DanielMing123]

We closely examine both evaluation methods, and you are right. They are indeed different. Thank you for pointing out this huge mistake, and we appreciate your code review. We will retrain our model to study its boundaries.