MCG-NJU / SparseOcc

[ECCV 2024] Fully Sparse 3D Occupancy Prediction & RayIoU Evaluation Metric
https://arxiv.org/abs/2312.17118
Apache License 2.0
258 stars 20 forks source link

openocc数据集上复现结果问题 #44

Closed lzzzzzm closed 3 months ago

lzzzzzm commented 3 months ago

现在这个版本在openocc上好像有点问题,改了一下loss和class weights那里才能跑。但复现后,发现精度差距挺多的,这个具体是什么原因呢。


|     Class Names      | RayIoU@1 | RayIoU@2 | RayIoU@4 |
+----------------------+----------+----------+----------+
|         car          |  0.463   |  0.555   |  0.585   |
|        truck         |  0.372   |  0.488   |  0.547   |
|       trailer        |  0.183   |  0.250   |  0.337   |
|         bus          |  0.420   |  0.555   |  0.626   |
| construction_vehicle |  0.146   |  0.241   |  0.290   |
|       bicycle        |  0.189   |  0.214   |  0.221   |
|      motorcycle      |  0.181   |  0.210   |  0.219   |
|      pedestrian      |  0.271   |  0.323   |  0.340   |
|     traffic_cone     |  0.279   |  0.296   |  0.303   |
|       barrier        |  0.359   |  0.405   |  0.428   |
|  driveable_surface   |  0.452   |  0.548   |  0.656   |
|      other_flat      |  0.248   |  0.290   |  0.322   |
|       sidewalk       |  0.228   |  0.273   |  0.319   |
|       terrain        |  0.208   |  0.273   |  0.336   |
|       manmade        |  0.245   |  0.334   |  0.386   |
|      vegetation      |  0.168   |  0.259   |  0.344   |
+----------------------+----------+----------+----------+
|         MEAN         |  0.276   |  0.345   |  0.391   |
+----------------------+----------+----------+----------+
[2024-07-24 11:23:01,769][INFO] - --- Evaluation Results ---
[2024-07-24 11:23:01,769][INFO] - RayIoU: 0.3371
[2024-07-24 11:23:01,769][INFO] - RayIoU@1: 0.2757
[2024-07-24 11:23:01,769][INFO] - RayIoU@2: 0.3445
[2024-07-24 11:23:01,769][INFO] - RayIoU@4: 0.3912```
afterthat97 commented 3 months ago

就是这样,没啥问题啊,我们论文里是在occ3d做的,不是openocc

lzzzzzm commented 3 months ago

但理论上,同一个算法在openocc和occ3d上的结果差异应该比较小吧,甚至应该openocc比occ3d要高,因为没有others类别了。这都差了3个点了,我以为是我改的哪里出了问题

afterthat97 commented 3 months ago

这俩数据集具体还是有一些区别的,你可以对比一下GT。 实际上occ3d跑出来结果还是要高一些,你可以跑 r50_nuimg_704x256_8f.py 这个setting试试

lzzzzzm commented 3 months ago

好的,感谢