Evaluate Result precision

[vehxianfish]

Hi, thanks to open source excellent work. But after training, the result is not well as the paper said. The result is as follows: I guess it was some fault when I installed or trained. my environment is as follows:

ubuntu 18.04
pytorch=1.10.0
spconv=2.1.25

[vehxianfish]

Otherwise, the performance of provided training log on the validation dataset is not equal to the result in README.md. whether is an uploaded error?

[sankin97]

Please provide your training log. We only use 2GPUs and a total batch size of 4 due to the limited data of KITTI. The results in README are the mean AP@R40 among three difficulties for each class.

[vehxianfish]

Thanks for your quick reply. the log is in this link We use 3 GPUs and a total batch size of 48 to validate the result quickly. But, in general, the result is better if the batch size is larger. And at least it can't be worse. So, I want to know what step is at fault.

[Marsjunwang]

Thanks for the excellent work.I also meet the performance degradtion. I just use the weights provided in the project to evaluate the kitti val dataset,but the result is not as good as paper.The result is as fellows: 2023-05-18 12:17:58,491 INFO Car AP@0.70, 0.70, 0.70: bbox AP:90.5662, 89.3980, 89.0797 bev AP:89.7979, 87.8464, 87.5922 3d AP:88.9249, 83.4915, 78.8589 aos AP:90.56, 89.33, 88.96 Car AP_R40@0.70, 0.70, 0.70: bbox AP:96.2178, 94.3393, 94.0154 bev AP:92.9286, 90.6000, 90.0815 3d AP:91.8647, 84.7695, 82.7189 aos AP:96.20, 94.25, 93.86 Car AP@0.70, 0.50, 0.50: bbox AP:90.5662, 89.3980, 89.0797 bev AP:90.5922, 89.3992, 89.1303 3d AP:90.5922, 89.3858, 89.1040 aos AP:90.56, 89.33, 88.96 Car AP_R40@0.70, 0.50, 0.50: bbox AP:96.2178, 94.3393, 94.0154 bev AP:96.2808, 94.4365, 94.3368 3d AP:96.2665, 94.4014, 94.2717 aos AP:96.20, 94.25, 93.86 Pedestrian AP@0.50, 0.50, 0.50: bbox AP:77.3539, 73.8375, 69.4597 bev AP:71.9948, 65.6862, 63.1394 3d AP:67.6584, 62.8511, 58.3649 aos AP:72.24, 68.22, 63.85 Pedestrian AP_R40@0.50, 0.50, 0.50: bbox AP:78.9637, 74.3189, 71.6092 bev AP:72.2779, 66.1497, 63.1365 3d AP:69.1400, 62.9257, 58.9703 aos AP:73.23, 68.25, 65.21 Pedestrian AP@0.50, 0.25, 0.25: bbox AP:77.3539, 73.8375, 69.4597 bev AP:82.2400, 76.4459, 74.7539 3d AP:82.2339, 76.3862, 74.7103 aos AP:72.24, 68.22, 63.85 Pedestrian AP_R40@0.50, 0.25, 0.25: bbox AP:78.9637, 74.3189, 71.6092 bev AP:83.3106, 79.2258, 76.3669 3d AP:83.3058, 79.1496, 76.2804 aos AP:73.23, 68.25, 65.21 Cyclist AP@0.50, 0.50, 0.50: bbox AP:89.5139, 82.8493, 77.1943 bev AP:93.2137, 74.1967, 72.0911 3d AP:87.0630, 73.2773, 70.7729 aos AP:89.34, 82.15, 76.42 Cyclist AP_R40@0.50, 0.50, 0.50: bbox AP:95.0707, 83.0192, 80.4710 bev AP:94.5547, 76.6651, 73.5429 3d AP:91.8386, 75.3479, 71.2423 aos AP:94.81, 82.28, 79.58 Cyclist AP@0.50, 0.25, 0.25: bbox AP:89.5139, 82.8493, 77.1943 bev AP:95.0232, 78.8821, 73.6608 3d AP:95.0232, 78.8821, 73.6608 aos AP:89.34, 82.15, 76.42 Cyclist AP_R40@0.50, 0.25, 0.25: bbox AP:95.0707, 83.0192, 80.4710 bev AP:95.7612, 79.0532, 76.3431 3d AP:95.7612, 79.0532, 76.3431 aos AP:94.81, 82.28, 79.58 log_eval_20230518-113651.txt I use a GPU 1060 and ubuntu 20.4

[sankin97]

Thanks for the excellent work.I also meet the performance degradtion. I just use the weights provided in the project to evaluate the kitti val dataset,but the result is not as good as paper.The result is as fellows: 2023-05-18 12:17:58,491 INFO Car AP@0.70, 0.70, 0.70: bbox AP:90.5662, 89.3980, 89.0797 bev AP:89.7979, 87.8464, 87.5922 3d AP:88.9249, 83.4915, 78.8589 aos AP:90.56, 89.33, 88.96 Car AP_R40@0.70, 0.70, 0.70: bbox AP:96.2178, 94.3393, 94.0154 bev AP:92.9286, 90.6000, 90.0815 3d AP:91.8647, 84.7695, 82.7189 aos AP:96.20, 94.25, 93.86 Car AP@0.70, 0.50, 0.50: bbox AP:90.5662, 89.3980, 89.0797 bev AP:90.5922, 89.3992, 89.1303 3d AP:90.5922, 89.3858, 89.1040 aos AP:90.56, 89.33, 88.96 Car AP_R40@0.70, 0.50, 0.50: bbox AP:96.2178, 94.3393, 94.0154 bev AP:96.2808, 94.4365, 94.3368 3d AP:96.2665, 94.4014, 94.2717 aos AP:96.20, 94.25, 93.86 Pedestrian AP@0.50, 0.50, 0.50: bbox AP:77.3539, 73.8375, 69.4597 bev AP:71.9948, 65.6862, 63.1394 3d AP:67.6584, 62.8511, 58.3649 aos AP:72.24, 68.22, 63.85 Pedestrian AP_R40@0.50, 0.50, 0.50: bbox AP:78.9637, 74.3189, 71.6092 bev AP:72.2779, 66.1497, 63.1365 3d AP:69.1400, 62.9257, 58.9703 aos AP:73.23, 68.25, 65.21 Pedestrian AP@0.50, 0.25, 0.25: bbox AP:77.3539, 73.8375, 69.4597 bev AP:82.2400, 76.4459, 74.7539 3d AP:82.2339, 76.3862, 74.7103 aos AP:72.24, 68.22, 63.85 Pedestrian AP_R40@0.50, 0.25, 0.25: bbox AP:78.9637, 74.3189, 71.6092 bev AP:83.3106, 79.2258, 76.3669 3d AP:83.3058, 79.1496, 76.2804 aos AP:73.23, 68.25, 65.21 Cyclist AP@0.50, 0.50, 0.50: bbox AP:89.5139, 82.8493, 77.1943 bev AP:93.2137, 74.1967, 72.0911 3d AP:87.0630, 73.2773, 70.7729 aos AP:89.34, 82.15, 76.42 Cyclist AP_R40@0.50, 0.50, 0.50: bbox AP:95.0707, 83.0192, 80.4710 bev AP:94.5547, 76.6651, 73.5429 3d AP:91.8386, 75.3479, 71.2423 aos AP:94.81, 82.28, 79.58 Cyclist AP@0.50, 0.25, 0.25: bbox AP:89.5139, 82.8493, 77.1943 bev AP:95.0232, 78.8821, 73.6608 3d AP:95.0232, 78.8821, 73.6608 aos AP:89.34, 82.15, 76.42 Cyclist AP_R40@0.50, 0.25, 0.25: bbox AP:95.0707, 83.0192, 80.4710 bev AP:95.7612, 79.0532, 76.3431 3d AP:95.7612, 79.0532, 76.3431 aos AP:94.81, 82.28, 79.58 log_eval_20230518-113651.txt I use a GPU 1060 and ubuntu 20.4

This issue may be caused by a specific code line located at line. To ensure consistency with the model training, we have made updates and are now using cv2.

[Marsjunwang]

Thanks for your reply. Now i am sure the skimage.io.imread is different from the cv2.imread.The link you provided maybe need to include the skio.imread in the get_image() funtion not just get_image_shape() function. Thank you for you quick reply again.The new evaluation log is as fellows for anyone who may need: log_eval_20230604-084754.txt

[russellyq]

Hi @sankin97 ,

this line (https://github.com/PJLab-ADG/LoGoNet/blob/efd894a2825072959beae26c887cd4073d225fe2/detection/al3d_det/datasets/kitti/kitti_dataset.py#L119) is still "skio.imread". Have you updated to cv2 ?