I can't reach the car moderate precision which I trained for 80 epochs

[Eaton2022]

I trained the CasA-V for 80 epochs, 3 classes., while I found I can't reach car moderate 3D precision which recall is 11 and by the way ,the other two classes reached the precision as you write in paper. I think my batchsize is too small, it's two. I wonder can I use two batchsize to obtain high precision? How to reset learning rate? Here is the result. * EPOCH 80 EVALUATION *** bbox AP:90.7547, 89.7766, 89.5010 bev AP:90.2147, 88.2992, 88.0006 3d AP:89.6871, 79.6853, 79.2249 aos AP:90.74, 89.67, 89.31 Car AP_R40@0.70, 0.70, 0.70: bbox AP:96.8148, 95.5741, 93.2506 bev AP:93.6572, 89.8203, 87.5100 3d AP:92.9158, 84.1482, 83.7333

[hailanyi]

This may be caused by the high score threshold, you can use lower score threshold (0.7->0.6 or 0.7->0.65) in the CasA-V.yaml to test your model.

    POST_PROCESSING:
        RECALL_THRESH_LIST: [0.3, 0.5, 0.7]
        SCORE_THRESH: 0.6  # tuning the score threshold here to obtain good results 
        OUTPUT_RAW_SCORE: False

[yiri2022]

Why is the network so sensitive to the threshold? By adjusting this threshold, I also achieved a similar AP to the paper.

bbox AP:98.1848, 89.9826, 89.6064 bev AP:90.3334, 88.6443, 88.1494 3d AP:89.6972, 86.2327, 79.2771 aos AP:98.13, 89.89, 89.46 Car AP_R40@0.70, 0.70, 0.70: bbox AP:98.9700, 95.7557, 93.3304 bev AP:93.7706, 92.0452, 89.7328 3d AP:92.8850, 86.1569, 83.7628 aos AP:98.93, 95.63, 93.15 Car AP@0.70, 0.50, 0.50: bbox AP:98.1848, 89.9826, 89.6064 bev AP:98.2297, 89.9412, 89.6159 3d AP:98.2073, 89.9306, 89.5879 aos AP:98.13, 89.89, 89.46 Car AP_R40@0.70, 0.50, 0.50: bbox AP:98.9700, 95.7557, 93.3304 bev AP:98.9805, 95.7699, 93.3410 3d AP:98.8878, 95.7337, 93.3036 aos AP:98.93, 95.63, 93.15 Pedestrian AP@0.50, 0.50, 0.50: bbox AP:80.2600, 73.4548, 71.9500 bev AP:71.2903, 68.6034, 61.9450 3d AP:70.3801, 67.5366, 60.8072 aos AP:77.47, 70.51, 68.57 Pedestrian AP_R40@0.50, 0.50, 0.50: bbox AP:81.3972, 75.1650, 70.4979 bev AP:74.0048, 68.8206, 62.4292 3d AP:72.8491, 66.2445, 59.8847 aos AP:78.36, 71.88, 67.02 Pedestrian AP@0.50, 0.25, 0.25: bbox AP:80.2600, 73.4548, 71.9500 bev AP:82.9656, 80.9756, 73.6741 3d AP:83.0630, 80.9756, 73.6741 aos AP:77.47, 70.51, 68.57 Pedestrian AP_R40@0.50, 0.25, 0.25: bbox AP:81.3972, 75.1650, 70.4979 bev AP:86.9962, 82.2175, 75.5535 3d AP:87.0576, 82.2166, 75.5526 aos AP:78.36, 71.88, 67.02 Cyclist AP@0.50, 0.50, 0.50: bbox AP:89.0014, 78.5029, 78.0384 bev AP:87.7107, 74.6787, 67.6883 3d AP:86.5453, 72.9849, 66.3992 aos AP:88.70, 78.05, 77.54 Cyclist AP_R40@0.50, 0.50, 0.50: bbox AP:94.9959, 79.1564, 76.5699 bev AP:93.3758, 75.2013, 70.7382 3d AP:91.8668, 71.9886, 67.4627 aos AP:94.58, 78.67, 76.05 Cyclist AP@0.50, 0.25, 0.25: bbox AP:89.0014, 78.5029, 78.0384 bev AP:88.0737, 75.3276, 74.3367 3d AP:88.0737, 75.3276, 74.3367 aos AP:88.70, 78.05, 77.54 Cyclist AP_R40@0.50, 0.25, 0.25: bbox AP:94.9959, 79.1564, 76.5699 bev AP:93.8242, 75.7949, 73.0799 3d AP:93.8242, 75.7949, 73.0799 aos AP:94.58, 78.67, 76.05

[jlqzzz]

@yiri2022 Hi, what is the threshold set to achieve similar results in the paper? 0.65 ？ 0.6？

[Eaton2022]

:72.8491, 66.2445, 59.8847

0.6. But I can get the result which threshold is 0.7 when I set batchsize is 2 per GPU.

[hailanyi]

@jlqzzz , @Eaton2022 @yiri2022 ,thanks for discuss this problem, I have updated this code. Now, one can easily attain a good performance by training the default config. We show a CasA-V results reproduced by 2080 GPU with spconv2 : 2022-11-29 02:52:34,960 INFO Average predicted number of objects(3769 samples): 4.774 2022-11-29 02:52:45,250 INFO Car AP@0.70, 0.70, 0.70: bbox AP:98.5548, 89.9622, 89.6169 bev AP:90.3222, 88.6719, 88.2400 3d AP:89.6201, 86.6608, 79.3748 aos AP:98.51, 89.88, 89.47 Car AP_R40@0.70, 0.70, 0.70: bbox AP:99.2572, 95.8205, 93.3752 bev AP:96.0735, 92.1997, 89.8395 3d AP:93.0469, 86.4311, 83.9911 aos AP:99.23, 95.70, 93.20