search

djiajunustc / H-23D_R-CNN

Apache License 2.0
63 stars 4 forks source link

Unexpected result on KITTI validation set #1

Open wei-tim opened 3 years ago

wei-tim commented 3 years ago

Hi,

thanks for sharing the code!

When I train the H-23D_R-CNN on KITTI dataset, I can not get the same results as reported in your paper. So I wonder whether I did something wrong...

I trained the model with yaml file ‘hh3d_rcnn_car.yaml’ with standard KITTI training set. However, I only obtained the following result on validation set:

2021-08-10 22:19:12,492 INFO Car AP@0.70, 0.70, 0.70: bbox AP:90.7384, 89.6182, 89.0859 bev AP:90.2286, 88.1226, 87.2080 3d AP:89.3387, 79.3252, 78.6396 aos AP:90.73, 89.47, 88.82 Car AP_R40@0.70, 0.70, 0.70: bbox AP:96.3041, 94.7434, 92.4225 bev AP:93.3221, 89.3447, 88.8444 3d AP:91.9288, 83.2978, 80.7666 aos AP:96.28, 94.54, 92.11 Car AP@0.70, 0.50, 0.50: bbox AP:90.7384, 89.6182, 89.0859 bev AP:90.7384, 89.6770, 89.2395 3d AP:90.7384, 89.6559, 89.1968 aos AP:90.73, 89.47, 88.82 Car AP_R40@0.70, 0.50, 0.50: bbox AP:96.3041, 94.7434, 92.4225 bev AP:96.3670, 95.0878, 94.6161 3d AP:96.3462, 95.0288, 92.6059 aos AP:96.28, 94.54, 92.11

3D ap moderate can only achieve 79.32, much lower than the reported score: 85.20...

Many thanks in advance, looking forward to your reply.

Best, Tim

djiajunustc commented 3 years ago

Hi，

You can try it again. Have you ever tested other epochs?

wei-tim commented 3 years ago

Hi,

thanks for your reply.

What do you mean for 'try again'? Should I train the model again and with same parameters?

I have already check all epochs between 70 and 80 epochs, but the results are similar.

Best, Tim

djiajunustc commented 3 years ago

Hi,

I mean that maybe you should train it again. My results are as follows:

Car AP@0.70, 0.70, 0.70: bbox AP:90.7833, 89.8625, 89.3937 bev AP:90.3749, 88.4831, 87.6475 3d AP:89.6334, 85.2028, 79.0819 aos AP:90.77, 89.76, 89.21 Car AP_R40@0.70, 0.70, 0.70: bbox AP:96.4862, 95.2911, 92.8600 bev AP:93.5535, 91.5020, 89.2045 3d AP:92.5864, 85.6215, 83.0278 aos AP:96.46, 95.15, 92.63 Car AP@0.70, 0.50, 0.50: bbox AP:90.7833, 89.8625, 89.3937 bev AP:90.7619, 89.8045, 89.4118 3d AP:90.7619, 89.7925, 89.3798 aos AP:90.77, 89.76, 89.21 Car AP_R40@0.70, 0.50, 0.50: bbox AP:96.4862, 95.2911, 92.8600 bev AP:96.5058, 95.3272, 94.9229 3d AP:96.4864, 95.2874, 94.8092 aos AP:96.46, 95.15, 92.63

I'll add the link in README to enable others to download my model.

Could you provide the performance if you set ROI_HEAD.NMS_CONFIG.TEST.SCORE_THRESH as 0 and set ROI_HEAD.NMS_CONFIG.TEST.USE_FAST_NMS as False?

wei-tim commented 3 years ago

Hi,

I have modified the yaml file but get almost same results as before...

2021-08-11 15:09:52,975 INFO Car AP@0.70, 0.70, 0.70: bbox AP:90.7384, 89.6289, 89.0875 bev AP:90.2286, 88.1343, 87.1735 3d AP:89.3387, 79.3262, 78.6413 aos AP:90.73, 89.48, 88.82 Car AP_R40@0.70, 0.70, 0.70: bbox AP:96.2584, 94.7295, 92.3922 bev AP:93.2806, 89.3107, 88.8069 3d AP:91.8737, 83.2657, 80.7364 aos AP:96.24, 94.53, 92.08 Car AP@0.70, 0.50, 0.50: bbox AP:90.7384, 89.6289, 89.0875 bev AP:90.7384, 89.6748, 89.2504 3d AP:90.7384, 89.6548, 89.2115 aos AP:90.73, 89.48, 88.82 Car AP_R40@0.70, 0.50, 0.50: bbox AP:96.2584, 94.7295, 92.3922 bev AP:96.3228, 95.0477, 94.5852 3d AP:96.3010, 94.9908, 94.4228 aos AP:96.24, 94.53, 92.08

By the way, I use 8 GPUs with default batchsize=4, does it have an influence to the final result?

Best, Tim

djiajunustc commented 3 years ago

I also use 8GPUs with batchsize=4 per GPU. The results are weird. I'll pull this code and train it again.

wei-tim commented 3 years ago

That would be great, looking forward to your result ;-)

djiajunustc commented 3 years ago

I have finished the README now. Maybe you can also train it again with the docker image I provide.

djiajunustc commented 3 years ago

Hi @wei-tim , I have trained the model with the released code. I can reproduce the results. This is my log file: https://drive.google.com/file/d/1OE_K_m_ugztewieCsZ9zasZOZlsToW8M/view?usp=sharing. Hope this can help you.

Eaphan commented 3 years ago

@wei-tim Do you set USE_ROAD_PLANE=True, according to my experiments this parameter is crucial.

Eaphan commented 3 years ago

Hi @wei-tim , I have trained the model with the released code. I can reproduce the results. This is my log file: https://drive.google.com/file/d/1OE_K_m_ugztewieCsZ9zasZOZlsToW8M/view?usp=sharing. Hope this can help you.

Thanks for your work. Would you mind to share your reproduced results while USE_ROAD_PLANE is set to False.