search

Laiqingsi / CLOCs_LQS

An implementation of CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detection.
43 stars 5 forks source link

AP does not improve through fusion #8

Open kyrie-louy opened 2 years ago

kyrie-louy commented 2 years ago

Hi @Laiqingsi, thanks for your great work. It helps me a lot!

However, I found that the AP improvement by fusion is very low (even degradation in some cases) compared to original CLOCs implementation. For example, when I train the fusion model with my own 2d detection results from Retinanet in Detectron2, it doesn't gain anything from fusion. Oppositely, I can get a 3%+ improvement in the original implementation.

Results based on your code:

LiDAR-only:
Car AP@0.70, 0.70, 0.70:
bbox AP:90.8318, 90.0370, 89.2701
bev  AP:90.0336, 88.1284, 86.8221
3d   AP:88.6435, 78.5305, 77.4470
aos  AP:90.82, 89.93, 89.11
Car AP_R40@0.70, 0.70, 0.70:
bbox AP:96.1435, 94.8264, 92.1872
bev  AP:92.8031, 90.3776, 88.0352
3d   AP:90.6124, 81.9148, 79.0518
aos  AP:96.13, 94.69, 92.00

CLOCs:
Car AP@0.70, 0.70, 0.70:
bbox AP:94.2373, 90.3053, 89.6129
bev  AP:89.8845, 88.1735, 87.0562
3d   AP:88.3369, 78.4858, 77.3919
aos  AP:94.20, 90.22, 89.47
Car AP_R40@0.70, 0.70, 0.70:
bbox AP:97.5323, 94.8859, 93.8404
bev  AP:93.5442, 90.4057, 87.8485
3d   AP:90.2475, 81.7896, 78.7422
aos  AP:97.50, 94.77, 93.64

It's quite strange as I can't find an obvious difference between your code and the original implementation. I wonder that have you encountered such an issue when evaluating different 2d predictions? What may be the possible reason?

Notes: I've also tried to increase the 3D prediction number to 70400 for the second model, but got a similar observation.

Laiqingsi commented 2 years ago

I am sorry about this situation. I have tried several different 2D detection methods through mmdet, while they works well. I've never encountered a situation like this before. You may verify that your 3D detection results are produced by OpenPCDet and that your 2D detection results do not go through NMS. I'm afraid I can't make any additional suggestions. I hope you can resolve this issue.

shayari21 commented 1 year ago

Hello, i tried this technique to compute clocs and the results are really bad, Can you help me with this?

2023-02-06 14:49:50,138 INFO Car AP@0.70, 0.70, 0.70: bbox AP:4.2648, 4.2198, 3.2463 bev AP:4.1331, 4.0580, 3.1242 3d AP:4.0527, 3.8060, 2.9328 aos AP:4.26, 4.18, 3.21 Car AP_R40@0.70, 0.70, 0.70: bbox AP:4.4958, 4.3517, 3.2733 bev AP:4.1676, 4.1848, 3.1503 3d AP:4.0865, 3.6632, 2.6884 aos AP:4.49, 4.31, 3.24 Car AP@0.70, 0.50, 0.50: bbox AP:4.2648, 4.2198, 3.2463 bev AP:4.2682, 4.2643, 3.2814 3d AP:4.2680, 4.2537, 3.2734 aos AP:4.26, 4.18, 3.21 Car AP_R40@0.70, 0.50, 0.50: bbox AP:4.4958, 4.3517, 3.2733 bev AP:4.4994, 4.3975, 3.3087 3d AP:4.4992, 4.3866, 3.3007 aos AP:4.49, 4.31, 3.24