search

PRBonn / lidar-bonnetal

Semantic and Instance Segmentation of LiDAR point clouds for autonomous driving
http://semantic-kitti.org
MIT License
945 stars 205 forks source link

mIOU is quite low using your prediction files #33

Closed hagianga21 closed 4 years ago

hagianga21 commented 4 years ago

Hi, I tried submitting your prediction files (darknet53-1024 + KNN). The result is quite low. mIOU is 0.36. Why does it happen? Thank you so much and have a great day.

jbehley commented 4 years ago

Please ensure that you submit the predictions to the correct phase of the competition. We have a "multiple scans" phase, where all classes with distinguishing moving and non-moving need to be predicted, and a "single scans" phase, where this is not accounted for. The provided prediction files are for the "single scan" phase.

Hope that clarifies the discrepency.

hagianga21 commented 4 years ago

Hi, I submitted to "single scans" phase, which is the right place. Is darknet53-1024 + KNN has the highest score? I can double-check it again.

jbehley commented 4 years ago

darknet53+knn is the best model. The 1024 and 512 models used a reduced resolution of the range image projection. As far as I remember, we did submit exactly this file. It can be that there is a difference of 1-2% due to a bugfix in the evaluation script, but there should not be a difference of over 15%.

I will also check again that there is not a problem on our side.

hagianga21 commented 4 years ago

Hi I tried another time with darknet53+knn. The mIOU and Acc are 0.524 and 0.89. Thank you so much for helping me.

huixiancheng commented 3 years ago

This issue should again be noted, in my verification, darknet53-1024-knn.tar.gz in seq08 is 0.384, which is significantly lower than reported in the paper. QQ截图20210707165022

jbehley commented 3 years ago

Did you downsample the range image to the size 1024x64? And note that the numbers in Table I are from the test set. But these should be quite similar.