search

xieKKKi / MotionBEV

A simple yet effective framework for LiDAR-based moving object segmentation.
MIT License
81 stars 5 forks source link

About IoU on semanticKITTI testing dataset #1

Closed liamlin5566 closed 10 months ago

liamlin5566 commented 11 months ago

Hi, thanks for sharing your great work. I noticed that your testing result on semanticKITTI leaderboard is 75.8, which is higher than the result on the paper. Do you use KITTI-Road dataset for training, or use other hyperparameter setting to achieve the result? BTW, do you provide the pretrained models of 75.8? Is "MotionBEV-kitti-road-test-74.88.pt" the file name of the pretrained weight of 75.8? Thanks.

xieKKKi commented 11 months ago

Hi, thanks for sharing your great work. I noticed that your testing result on semanticKITTI leaderboard is 75.8, which is higher than the result on the paper. Do you use KITTI-Road dataset for training, or use other hyperparameter setting to achieve the result? BTW, do you provide the pretrained models of 75.8? Is "MotionBEV-kitti-road-test-74.88.pt" the file name of the pretrained weight of 75.8? Thanks.

Hi, @liamlin5566 . Thank you for your interest in our work! The result on the semanticKITTI leaderboard (IoU 75.8%) was achieved by using KITTI road data for training and combining predictions from a semantic segmentation network to remove immovable objects. The "MotionBEV-kitti-road-test-74.88.pt" model is trained on both SemanticKITTI and KITTI road data, achieving an IoU of 74.88% without relying on semantic information. The results reported in our paper do not use KITTI road data because the hidden SemanticKITTI test sequence 20 fully overlaps with the KITTI road sequence 41 used for validation.

liamlin5566 commented 11 months ago

Thanks for your reply. I also notice that you will drop some static frames in MotionBEV-semantickitti.yaml, and I wonder whether "MotionBEV-kitti-road-test-74.88.pt" is also trained with dropping static frames. Thanks.

xieKKKi commented 11 months ago

Thanks for your reply. I also notice that you will drop some static frames in MotionBEV-semantickitti.yaml, and I wonder whether "MotionBEV-kitti-road-test-74.88.pt" is also trained with dropping static frames. Thanks.

We do drop static frames when training with SemanticKITTI data, but do not drop static frames when training with additional KITTI road data.

liamlin5566 commented 11 months ago

OK, thanks for answering my questions?

liamlin5566 commented 11 months ago

Hi @xieKKKi , sorry for disturbing again. I ecounter unstable val performance when trained with additional KITTI-Road data and dropping the static frames in original semanticKITTI dataset. In https://github.com/xieKKKi/MotionBEV/issues/1#issuecomment-1855659440, do you means that you do not drop the frames in original semanticKITTI dataset? Thaks

xieKKKi commented 11 months ago

Hi @xieKKKi , sorry for disturbing again. I ecounter unstable val performance when trained with additional KITTI-Road data and dropping the static frames in original semanticKITTI dataset. In #1 (comment), do you means that you do not drop the frames in original semanticKITTI dataset? Thaks

I set "drop_few_static_frames" in "MotionBEV-semantickitti.yaml" as True when training with SemanticKITTI data, and set it as False when training with both SemanticKITTI data and additional KITTI road data. Can you provide more details about the unstable val performance? Did you train from scratch?

liamlin5566 commented 11 months ago

Thank you for replying. I train from scratch. The val moving IoU can reach ~73% while the next epoch drops to ~65% although the change of training loss is small. I would try to train the model without dropping any frames.

liamlin5566 commented 10 months ago

Sorry for late replying. The val performance is more stable without dropping any frames. And I also noticed that some stopped object is marked as moving (https://github.com/PRBonn/semantic-kitti-api/issues/80), which may causing some performance instability in the early training epoch. Thanks.