Question about validation results

[chenyuntc]

Thank you for your excellent work!

I download the prediction file darknet53-knn.tar.gz and run evaluate_semantics on sequence 08 and get:

Acc avg 0.842
IoU avg 0.375

I just wanna make sure, this is the SAME model as DarkNet53Seg in Table 2 of the SemanticKITTI paper, which got 49.9 mIoU in the test set (seq 11-21)

[chenyuntc]

It was actually my mistake, I was actually doing the evaluation on darkent53-512, the evaluation results on the darknet53-knn is actually

Acc avg 0.901
IoU avg 0.528

sorry for the trouble.

Again, I wanna confirm which is the one DarkNet53Seg in Table 2?

[jbehley]

Thanks for your interest in our work.

Table 2 is evaluated on the test set (seq 11 - seq 21). [See also the caption of Table 2: Single scan results (19 classes) for all baselines on sequences 11 to 21 (test set). All methods were trained on sequences 00 to 10, except for sequence 08 which is used as validation set.]

However, the model in Table 2 is Darknet53 without KNN postprocessing. Therefore, if you want the model of Table 2 you have to download darknet53.tar.gz. Darknet53 + KNN is getting 52.2 mIoU on the testset, as shown in Table I of our follow-up paper RangeNet++: Fast and Accurate LiDAR Semantic Segmentation.

Hope that clarifies everything.

[chenyuntc]

Thank you for your clarification!!

[Sabze]

Hi, did you also use CRF in your model Darknet53 + KNN (that got 52.2 mIoU in Table I in your paper RangeNet++: Fast and Accurate LiDAR Semantic Segmentation )?

[tano297]

No CRF in the RangeNet++ models

[Sabze]

Thanks for the quick reply!

[chengdazhi]

Hi, I tried to evaluate the predictions darknet53-knn.tar.gz downloaded from readme on the validation set (08 sequence with 4071 frames)

python ./evaluate_iou.py -d semantic_kitti -p darknet53-knn --split valid

but I only got: Acc avg 0.718 IoU avg 0.454

instead of Acc avg 0.901 IoU avg 0.528

As posted by @chenyuntc. Could anyone please share your script for eval on val set?

[chengdazhi]

I found the script evaluate_semantic.py you mentioned is at semantic kitti api repo. And using evaluate_semantic.py I can reproduce your results. Now the question is: why would these two scripts produce different results?

evaluate_iou.py produces: Acc avg 0.718 IoU avg 0.454 IoU class 1 [car] = 0.848 IoU class 2 [bicycle] = 0.250 IoU class 3 [motorcycle] = 0.471 IoU class 4 [truck] = 0.407 IoU class 5 [other-vehicle] = 0.255 IoU class 6 [person] = 0.452 IoU class 7 [bicyclist] = 0.629 IoU class 8 [motorcyclist] = 0.000 IoU class 9 [road] = 0.749 IoU class 10 [parking] = 0.465 IoU class 11 [sidewalk] = 0.580 IoU class 12 [other-ground] = 0.002 IoU class 13 [building] = 0.657 IoU class 14 [fence] = 0.542 IoU class 15 [vegetation] = 0.397 IoU class 16 [trunk] = 0.529 IoU class 17 [terrain] = 0.463 IoU class 18 [pole] = 0.532 IoU class 19 [traffic-sign] = 0.400

While evaluate_semantic.py outputs: Acc avg 0.901 IoU avg 0.528 IoU class 1 [car] = 0.910 IoU class 2 [bicycle] = 0.250 IoU class 3 [motorcycle] = 0.471 IoU class 4 [truck] = 0.407 IoU class 5 [other-vehicle] = 0.255 IoU class 6 [person] = 0.452 IoU class 7 [bicyclist] = 0.629 IoU class 8 [motorcyclist] = 0.000 IoU class 9 [road] = 0.938 IoU class 10 [parking] = 0.465 IoU class 11 [sidewalk] = 0.819 IoU class 12 [other-ground] = 0.002 IoU class 13 [building] = 0.858 IoU class 14 [fence] = 0.542 IoU class 15 [vegetation] = 0.842 IoU class 16 [trunk] = 0.529 IoU class 17 [terrain] = 0.727 IoU class 18 [pole] = 0.532 IoU class 19 [traffic-sign] = 0.400

[iris0329]

IoU avg 0.454

Hi, @chengdazhi, Same as your result, I also got IoU avg 0.454, rather than 0.528 on the valid sequence 08, when using darknet53-knn prediction result the authors provided.

it was solved, because I found that, my Pytorch Version is 1.4, so I change the long type into float type according to the advice in this issue when running evaluate_iou.py

however, long type is 64 bit integer, float is 32 bit. so if change long type to double when using Pytorch 1.4. the ultimate iou is same as the result computed using script evaluate_semantic.py` at semantic kitti api repo.