search

lzccccc / SMOKE

SMOKE: Single-Stage Monocular 3D Object Detection via Keypoint Estimation
MIT License
695 stars 177 forks source link

results compared to centernet #19

Open qishuai913 opened 4 years ago

qishuai913 commented 4 years ago

Hi, thank you for your attention.I trained 3712 training examples and evaluated 3769 validation examples according to the parameters given in your paper.I got results worse than centernet.Does anyone get similar results with me?How to improve it? centernet: car_detection AP: 96.429062 87.383652 78.831161 car_orientation AP: 93.708160 83.642677 75.108299 pedestrian_detection AP: 69.460915 60.578434 52.195190 pedestrian_orientation AP: 57.431622 49.492664 42.333351 cyclist_detection AP: 73.471001 48.506691 41.843220 cyclist_orientation AP: 65.426979 43.168201 37.650307 car_detection_ground AP: 31.262831 29.248146 25.693661 pedestrian_detection_ground AP: 21.528582 20.380455 16.530041 cyclist_detection_ground AP: 21.939928 13.853647 13.297730 car_detection_3d AP: 17.370939 17.168695 15.270061 pedestrian_detection_3d AP: 20.790129 19.560652 15.860862 cyclist_detection_3d AP: 21.344902 13.315850 12.881376 SMOKE: car_detection AP: 86.502937 77.495712 68.934151 car_orientation AP: 86.208595 77.035370 68.207565 pedestrian_detection AP: 62.393944 54.208607 46.607456 pedestrian_orientation AP: 43.747810 37.509506 32.416935 cyclist_detection AP: 41.442078 30.185410 29.691633 cyclist_orientation AP: 23.559668 16.461676 16.320688 car_detection_ground AP: 22.207165 18.516039 16.109665 pedestrian_detection_ground AP: 8.190163 6.635043 6.192138 cyclist_detection_ground AP: 1.415183 0.826446 0.826446 car_detection_3d AP: 16.936316 14.226256 13.635677 pedestrian_detection_3d AP: 6.768989 6.250000 5.621096 cyclist_detection_3d AP: 1.376730 0.826446 0.826446

dingfuzhou commented 4 years ago

Whether you exactly used the same evaluation script?

malianghui commented 3 years ago

Hi, thank you for your attention.I trained 3712 training examples and evaluated 3769 validation examples according to the parameters given in your paper.I got results worse than centernet.Does anyone get similar results with me?How to improve it? centernet: car_detection AP: 96.429062 87.383652 78.831161 car_orientation AP: 93.708160 83.642677 75.108299 pedestrian_detection AP: 69.460915 60.578434 52.195190 pedestrian_orientation AP: 57.431622 49.492664 42.333351 cyclist_detection AP: 73.471001 48.506691 41.843220 cyclist_orientation AP: 65.426979 43.168201 37.650307 car_detection_ground AP: 31.262831 29.248146 25.693661 pedestrian_detection_ground AP: 21.528582 20.380455 16.530041 cyclist_detection_ground AP: 21.939928 13.853647 13.297730 car_detection_3d AP: 17.370939 17.168695 15.270061 pedestrian_detection_3d AP: 20.790129 19.560652 15.860862 cyclist_detection_3d AP: 21.344902 13.315850 12.881376 SMOKE: car_detection AP: 86.502937 77.495712 68.934151 car_orientation AP: 86.208595 77.035370 68.207565 pedestrian_detection AP: 62.393944 54.208607 46.607456 pedestrian_orientation AP: 43.747810 37.509506 32.416935 cyclist_detection AP: 41.442078 30.185410 29.691633 cyclist_orientation AP: 23.559668 16.461676 16.320688 car_detection_ground AP: 22.207165 18.516039 16.109665 pedestrian_detection_ground AP: 8.190163 6.635043 6.192138 cyclist_detection_ground AP: 1.415183 0.826446 0.826446 car_detection_3d AP: 16.936316 14.226256 13.635677 pedestrian_detection_3d AP: 6.768989 6.250000 5.621096 cyclist_detection_3d AP: 1.376730 0.826446 0.826446

@qishuai913 hello ! my result is bellow can you meet this problem?

100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3769/3769 [02:39<00:00, 23.57it/s] [2020-11-13 11:17:13,872] smoke.engine.inference INFO: Total run time: 0:02:39.928394 (0.042432579952451525 s / img per device, on 1 devices) [2020-11-13 11:17:13,872] smoke.engine.inference INFO: Model inference time: 0:02:23.100574 (0.037967782844185165 s / img per device, on 1 devices) [2020-11-13 11:17:13,872] smoke.data.datasets.evaluation.kitti.kitti_eval INFO: performing kitti detection evaluation: [2020-11-13 11:17:14,194] smoke.data.datasets.evaluation.kitti.kitti_eval INFO: Evaluate on KITTI dataset [2020-11-13 11:17:17,247] smoke.data.datasets.evaluation.kitti.kitti_eval INFO: PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek. ==> 1 page written on pedestrian_detection.pdf'. PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek. ==> 1 page written onpedestrian_orientation.pdf'. PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek. ==> 1 page written on cyclist_detection.pdf'. PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek. ==> 1 page written oncyclist_orientation.pdf'. PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek. ==> 1 page written on pedestrian_detection_ground.pdf'. PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek. ==> 1 page written oncyclist_detection_ground.pdf'. PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek. ==> 1 page written on pedestrian_detection_3d.pdf'. PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek. ==> 1 page written oncyclist_detection_3d.pdf'. Thank you for participating in our evaluation! Loading detections... number of files for evaluation: 3769 done. save /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train/plot/pedestrian_detection.txt pedestrian_detection AP: 0.000000 0.000000 0.000000 save /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train/plot/pedestrian_orientation.txt pedestrian_orientation AP: 0.000000 0.000000 0.000000 save /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train/plot/cyclist_detection.txt cyclist_detection AP: 0.000000 0.000000 0.000000 save /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train/plot/cyclist_orientation.txt cyclist_orientation AP: 0.000000 0.000000 0.000000 save /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train/plot/pedestrian_detection_ground.txt pedestrian_detection_ground AP: 0.000000 0.000000 0.000000 save /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train/plot/cyclist_detection_ground.txt cyclist_detection_ground AP: 0.000000 0.000000 0.000000 save /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train/plot/pedestrian_detection_3d.txt pedestrian_detection_3d AP: 0.000000 0.000000 0.000000 save /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train/plot/cyclist_detection_3d.txt cyclist_detection_3d AP: 0.000000 0.000000 0.000000 Your evaluation results are available at: /home/mlhui/project/SMOKE/tools/logs/inference/kitti_train

Gaozihui commented 3 years ago

und AP: 22.207165 18.516039 16.109665 pedestrian_detection_ground AP: 8.190163 6.635043 6.192138 cyclist_detection_ground AP: 1.415183 0.826

Hi, I am doing the same with you but encounter some problems. I am wandering if we can discuss about that. I would appreciate that if you can reply and you can connect with me by 414786527@qq.com

mrsempress commented 3 years ago

I run the code directly and evaluate on validation. The kitti 40 recall results are much lower.

Easy Moderate Hard
Car 6.74 / 12.17 4.35 / 8.09 4.02 / 7.65
Pedestrian 2.12 / 2.63 1.78 / 1.93 1.39 / 1.52
Cyclist 1.04 / 1.34 0.41 / 0.50 0.41 / 0.51

Who have some idea about my situation?