search

mpitropov / cadc_devkit

A devkit for the Canadian Adverse Driving Conditions (CADC) dataset.
http://cadcd.uwaterloo.ca/
Other
149 stars 36 forks source link

Very low AP with OpenPCDet cadc_support branch #31

Closed Blade6570 closed 2 years ago

Blade6570 commented 2 years ago

Hey,

I am using the cadc_support branch for the training and testing of 3d object detection on Lidar data. Qualitatively the results look fine to me but quantitatively it gives very poor values after 50 epochs with pointpillar default model:

Car AP@0.70, 0.70, 0.70:
bev  AP:26.2161, 26.2161, 26.2161
3d   AP:0.0045, 0.0045, 0.0045
Car AP_R40@0.70, 0.70, 0.70:
bev  AP:22.8053, 22.8053, 22.8053
3d   AP:0.0012, 0.0012, 0.0012
Car AP@0.70, 0.50, 0.50:
bev  AP:31.8633, 31.8633, 31.8633
3d   AP:0.0514, 0.0514, 0.0514
Car AP_R40@0.70, 0.50, 0.50:
bev  AP:29.4347, 29.4347, 29.4347
3d   AP:0.0141, 0.0141, 0.0141
Pedestrian AP@0.50, 0.50, 0.50:
bev  AP:1.2974, 1.2974, 1.2974
3d   AP:0.0005, 0.0005, 0.0005
Pedestrian AP_R40@0.50, 0.50, 0.50:
bev  AP:0.5754, 0.5754, 0.5754
3d   AP:0.0000, 0.0000, 0.0000
Pedestrian AP@0.50, 0.25, 0.25:
bev  AP:4.1253, 4.1253, 4.1253
3d   AP:0.0889, 0.0889, 0.0889
Pedestrian AP_R40@0.50, 0.25, 0.25:
bev  AP:3.6445, 3.6445, 3.6445
3d   AP:0.0524, 0.0524, 0.0524
Pickup_Truck AP@0.70, 0.70, 0.70:
bev  AP:5.6587, 5.6587, 5.6587
3d   AP:0.0000, 0.0000, 0.0000
Pickup_Truck AP_R40@0.70, 0.70, 0.70:
bev  AP:5.0133, 5.0133, 5.0133
3d   AP:0.0000, 0.0000, 0.0000
Pickup_Truck AP@0.50, 0.50, 0.50:
bev  AP:6.3875, 6.3875, 6.3875
3d   AP:0.0000, 0.0000, 0.0000
Pickup_Truck AP_R40@0.50, 0.50, 0.50:
bev  AP:5.6850, 5.6850, 5.6850
3d   AP:0.0000, 0.0000, 0.0000

There is something wrong with the evaluation script. Do you have any idea regarding this?

P.s. I am using the given deafault config files

Thanks, Soumya Tripathy

Blade6570 commented 2 years ago

removing this line , improves the 3d AP scores for cars. Now the the results are, 

 Car AP@0.70, 0.70, 0.70:
bev  AP:32.2144, 32.2144, 32.2144
3d   AP:19.0881, 19.0881, 19.0881
Car AP_R40@0.70, 0.70, 0.70:
bev  AP:29.3266, 29.3266, 29.3266
3d   AP:15.5297, 15.5297, 15.5297
Car AP@0.70, 0.50, 0.50:
bev  AP:42.0944, 42.0944, 42.0944
3d   AP:37.7694, 37.7694, 37.7694
Car AP_R40@0.70, 0.50, 0.50:
bev  AP:39.9449, 39.9449, 39.9449
3d   AP:35.2006, 35.2006, 35.2006
Pedestrian AP@0.50, 0.50, 0.50:
bev  AP:1.2862, 1.2862, 1.2862
3d   AP:0.9642, 0.9642, 0.9642
Pedestrian AP_R40@0.50, 0.50, 0.50:
bev  AP:0.5617, 0.5617, 0.5617
3d   AP:0.1328, 0.1328, 0.1328
Pedestrian AP@0.50, 0.25, 0.25:
bev  AP:4.2195, 4.2195, 4.2195
3d   AP:3.6293, 3.6293, 3.6293
Pedestrian AP_R40@0.50, 0.25, 0.25:
bev  AP:3.6386, 3.6386, 3.6386
3d   AP:3.0101, 3.0101, 3.0101
Pickup_Truck AP@0.70, 0.70, 0.70:
bev  AP:6.1766, 6.1766, 6.1766
3d   AP:4.7316, 4.7316, 4.7316
Pickup_Truck AP_R40@0.70, 0.70, 0.70:
bev  AP:5.4346, 5.4346, 5.4346
3d   AP:4.2141, 4.2141, 4.2141
Pickup_Truck AP@0.50, 0.50, 0.50:
bev  AP:6.6207, 6.6207, 6.6207
3d   AP:6.6164, 6.6164, 6.6164
Pickup_Truck AP_R40@0.50, 0.50, 0.50:
bev  AP:6.0628, 6.0628, 6.0628
3d   AP:6.0193, 6.0193, 6.0193

Other classes are still bad. I will investigate more but please have a look on the evaluation part.

Blade6570 commented 2 years ago

After removing the above mentioned line and training for few more epochs, I got the following results:

 Car AP@0.70, 0.70, 0.70:
bev  AP:51.4983, 51.4983, 51.4983
3d   AP:4.2109, 4.2109, 4.2109
Car AP_R40@0.70, 0.70, 0.70:
bev  AP:50.5045, 50.5045, 50.5045
3d   AP:2.5026, 2.5026, 2.5026
Car AP@0.70, 0.50, 0.50:
bev  AP:70.5230, 70.5230, 70.5230
3d   AP:46.6434, 46.6434, 46.6434
Car AP_R40@0.70, 0.50, 0.50:
bev  AP:70.2264, 70.2264, 70.2264
3d   AP:44.3507, 44.3507, 44.3507
Pedestrian AP@0.50, 0.50, 0.50:
bev  AP:37.0486, 37.0486, 37.0486
3d   AP:26.3659, 26.3659, 26.3659
Pedestrian AP_R40@0.50, 0.50, 0.50:
bev  AP:34.4007, 34.4007, 34.4007
3d   AP:22.8914, 22.8914, 22.8914
Pedestrian AP@0.50, 0.25, 0.25:
bev  AP:42.3045, 42.3045, 42.3045
3d   AP:41.4861, 41.4861, 41.4861
Pedestrian AP_R40@0.50, 0.25, 0.25:
bev  AP:40.1993, 40.1993, 40.1993
3d   AP:39.5600, 39.5600, 39.5600
Pickup_Truck AP@0.70, 0.70, 0.70:
bev  AP:23.3822, 23.3822, 23.3822
3d   AP:4.4535, 4.4535, 4.4535
Pickup_Truck AP_R40@0.70, 0.70, 0.70:
bev  AP:22.3618, 22.3618, 22.3618
3d   AP:3.8799, 3.8799, 3.8799
Pickup_Truck AP@0.50, 0.50, 0.50:
bev  AP:26.3057, 26.3057, 26.3057
3d   AP:24.6931, 24.6931, 24.6931
Pickup_Truck AP_R40@0.50, 0.50, 0.50:
bev  AP:25.0876, 25.0876, 25.0876
3d   AP:24.0102, 24.0102, 24.0102

Does this look close to your experiments? The values are very confusing to me.

mpitropov commented 2 years ago

Hi, I have been working on some updates to that branch. I also fixed that line you pointed out. The main issues are that the network is being trained on objects with low points and also we don't have difficulty levels.

I have created a new branch that has those fixes. https://github.com/mpitropov/OpenPCDet/pull/2 You will have to rerun the code to create the data infos. You should also use your old config files as I have updated those in the new branch.

Blade6570 commented 2 years ago

I was investigating on the older branch where it seems like after removing that line and finding a good checkpoint can give somewhat resonable results:

Car AP@0.70, 0.70, 0.70:
bev  AP:48.5314, 48.5314, 48.5314
3d   AP:26.0232, 26.0232, 26.0232
Car AP_R40@0.70, 0.70, 0.70:
bev  AP:47.5556, 47.5556, 47.5556
3d   AP:21.4210, 21.4210, 21.4210
Car AP@0.70, 0.50, 0.50:
bev  AP:63.7194, 63.7194, 63.7194
3d   AP:55.6401, 55.6401, 55.6401
Car AP_R40@0.70, 0.50, 0.50:
bev  AP:63.5322, 63.5322, 63.5322
3d   AP:54.9946, 54.9946, 54.9946
Pedestrian AP@0.50, 0.50, 0.50:
bev  AP:30.4946, 30.4946, 30.4946
3d   AP:23.0124, 23.0124, 23.0124
Pedestrian AP_R40@0.50, 0.50, 0.50:
bev  AP:27.2541, 27.2541, 27.2541
3d   AP:19.1644, 19.1644, 19.1644
Pedestrian AP@0.50, 0.25, 0.25:
bev  AP:39.0625, 39.0625, 39.0625
3d   AP:38.9852, 38.9852, 38.9852
Pedestrian AP_R40@0.50, 0.25, 0.25:
bev  AP:36.9300, 36.9300, 36.9300
3d   AP:36.8497, 36.8497, 36.8497
Pickup_Truck AP@0.70, 0.70, 0.70:
bev  AP:27.6949, 27.6949, 27.6949
3d   AP:17.2888, 17.2888, 17.2888
Pickup_Truck AP_R40@0.70, 0.70, 0.70:
bev  AP:26.1399, 26.1399, 26.1399
3d   AP:15.5157, 15.5157, 15.5157
Pickup_Truck AP@0.50, 0.50, 0.50:
bev  AP:30.3369, 30.3369, 30.3369
3d   AP:29.8558, 29.8558, 29.8558
Pickup_Truck AP_R40@0.50, 0.50, 0.50:
bev  AP:29.0019, 29.0019, 29.0019
3d   AP:28.7613, 28.7613, 28.7613

Now I am going to try the new branch. It would be great if you can share the AP numbers (or confirm to my numbers after I share them here) with Pointpillar's default model so that everyone can expect what to get.

thanks :) ~Soumya

Blade6570 commented 2 years ago

I have tested the new branch. Numbers look reasonable. I have used the NMS_THRESH=0.3 to generate this result. 

 INFO  Car AP@0.70, 0.70, 0.70:
bev  AP:61.7743, 44.9757, 30.0879
3d   AP:36.6587, 25.1438, 17.8584
Car AP_R40@0.70, 0.70, 0.70:
bev  AP:62.0301, 43.2371, 26.8890
3d   AP:33.3220, 20.0399, 11.9271
Car AP@0.70, 0.50, 0.50:
bev  AP:79.3418, 62.9923, 42.5869
3d   AP:75.5133, 57.5032, 38.1289
Car AP_R40@0.70, 0.50, 0.50:
bev  AP:81.3546, 63.3166, 40.5895
3d   AP:76.6057, 56.7458, 35.3378
Pedestrian AP@0.50, 0.50, 0.50:
bev  AP:20.1627, 17.8739, 13.9097
3d   AP:16.2375, 13.6925, 10.0548
Pedestrian AP_R40@0.50, 0.50, 0.50:
bev  AP:18.9590, 17.0558, 11.8507
3d   AP:14.9191, 12.3232, 7.9843
Pedestrian AP@0.50, 0.25, 0.25:
bev  AP:21.2510, 19.6710, 15.0994
3d   AP:21.2494, 19.6458, 15.0242
Pedestrian AP_R40@0.50, 0.25, 0.25:
bev  AP:20.1902, 18.8703, 13.5643
3d   AP:20.1888, 18.8405, 13.4032
Pickup_Truck AP@0.70, 0.70, 0.70:
bev  AP:49.6644, 38.5167, 25.2665
3d   AP:44.3345, 32.4599, 22.1387
Pickup_Truck AP_R40@0.70, 0.70, 0.70:
bev  AP:48.9444, 35.9102, 21.4969
3d   AP:42.5151, 29.4585, 17.4198
Pickup_Truck AP@0.50, 0.50, 0.50:
bev  AP:67.6431, 53.4258, 34.0099
3d   AP:67.3808, 52.0846, 33.2932
Pickup_Truck AP_R40@0.50, 0.50, 0.50:
bev  AP:68.5741, 51.9411, 31.3712
3d   AP:67.6994, 50.6481, 30.5807

What are your thoughts? ~soumya

mpitropov commented 2 years ago

That seems a bit lower than what I was able to get but it looks better than before. I trained a model for 80 epochs and I also used the new config file which could be why I'm seeing better results on my own model.

INFO  Car AP@0.70, 0.70, 0.70:
bev  AP:73.4323, 65.7316, 61.9929
3d   AP:60.0128, 52.1354, 45.2194
Car AP_R40@0.70, 0.70, 0.70:
bev  AP:75.6812, 65.9998, 60.3415
3d   AP:60.4139, 49.5878, 43.1697
Car AP@0.70, 0.50, 0.50:
bev  AP:89.1654, 84.9597, 77.5133
3d   AP:87.5146, 78.7879, 74.9445
Car AP_R40@0.70, 0.50, 0.50:
bev  AP:91.5318, 85.7023, 78.1675
3d   AP:89.5889, 80.6070, 74.4148
Pedestrian AP@0.50, 0.50, 0.50:
bev  AP:65.1790, 63.5803, 56.3103
3d   AP:54.6270, 52.1047, 46.1236
Pedestrian AP_R40@0.50, 0.50, 0.50:
bev  AP:65.1016, 63.4609, 55.7512
3d   AP:53.1146, 50.8713, 44.2206
Pedestrian AP@0.50, 0.25, 0.25:
bev  AP:72.0968, 71.1287, 63.5304
3d   AP:71.9909, 71.0744, 63.3729
Pedestrian AP_R40@0.50, 0.25, 0.25:
bev  AP:72.8834, 71.6423, 63.3628
3d   AP:72.8039, 71.5538, 63.2548
Pickup_Truck AP@0.70, 0.70, 0.70:
bev  AP:50.3788, 42.8631, 38.7765
3d   AP:35.2669, 32.7967, 30.3355
Pickup_Truck AP_R40@0.70, 0.70, 0.70:
bev  AP:48.2462, 40.9679, 35.7827
3d   AP:33.6950, 29.6397, 25.8266
Pickup_Truck AP@0.50, 0.50, 0.50:
bev  AP:60.8149, 52.7548, 46.7238
3d   AP:59.8392, 52.5672, 46.4322
Pickup_Truck AP_R40@0.50, 0.50, 0.50:
bev  AP:61.2663, 52.1240, 45.7929
3d   AP:60.8109, 51.7814, 45.1769
Blade6570 commented 2 years ago

Okay it looks very good! I tried on 15 epochs. So I will try tomorrow on larger epoch. Thanks a lot :) .

Blade6570 commented 2 years ago

Hey, I have now the results with 100 epochs and old configuration file. 

 INFO  Car AP@0.70, 0.70, 0.70:
bev  AP:63.7284, 48.6088, 33.1140
3d   AP:43.2645, 29.3240, 20.9878
Car AP_R40@0.70, 0.70, 0.70:
bev  AP:63.7238, 47.1576, 29.6224
3d   AP:41.1146, 26.4742, 15.7660
Car AP@0.70, 0.50, 0.50:
bev  AP:77.6752, 62.5597, 43.1742
3d   AP:74.7947, 59.2769, 39.5868
Car AP_R40@0.70, 0.50, 0.50:
bev  AP:78.9618, 62.4443, 40.4729
3d   AP:75.7317, 58.3173, 36.9342
Pedestrian AP@0.50, 0.50, 0.50:
bev  AP:26.3157, 28.8814, 22.3548
3d   AP:24.0628, 23.7043, 17.8993
Pedestrian AP_R40@0.50, 0.50, 0.50:
bev  AP:22.1788, 25.1451, 17.6847
3d   AP:19.4492, 19.0994, 12.6311
Pedestrian AP@0.50, 0.25, 0.25:
bev  AP:27.2807, 31.1527, 24.5435
3d   AP:27.2979, 31.1400, 24.5241
Pedestrian AP_R40@0.50, 0.25, 0.25:
bev  AP:23.1765, 27.6315, 20.0942
3d   AP:23.1876, 27.6194, 20.0716
Pickup_Truck AP@0.70, 0.70, 0.70:
bev  AP:31.8035, 34.7853, 23.9182
3d   AP:7.7757, 17.0099, 13.8665
Pickup_Truck AP_R40@0.70, 0.70, 0.70:
bev  AP:30.7286, 31.3477, 19.1380
3d   AP:6.7104, 11.8101, 7.6878
Pickup_Truck AP@0.50, 0.50, 0.50:
bev  AP:59.2373, 55.8125, 36.1916
3d   AP:56.4400, 53.1637, 34.6043
Pickup_Truck AP_R40@0.50, 0.50, 0.50:
bev  AP:59.2813, 55.4784, 34.0412
3d   AP:55.5685, 52.0831, 32.0187

accuracies for pickup truck are very bad. May be the new configuration file is the reason for good results. I am trying that now.

Blade6570 commented 2 years ago

Now I can confirm that everything works as expected.

Car AP@0.70, 0.70, 0.70:
bev  AP:71.8193, 64.8562, 56.9591
3d   AP:55.7935, 47.3126, 42.9851
Car AP_R40@0.70, 0.70, 0.70:
bev  AP:72.7658, 64.7421, 57.8623
3d   AP:54.0905, 45.0815, 39.1160
Car AP@0.70, 0.50, 0.50:
bev  AP:88.6435, 83.9837, 77.3653
3d   AP:86.9231, 78.4784, 73.9656
Car AP_R40@0.70, 0.50, 0.50:
bev  AP:90.7259, 85.2631, 77.8276
3d   AP:88.2412, 79.9746, 73.6169
Pedestrian AP@0.50, 0.50, 0.50:
bev  AP:65.5602, 63.8452, 56.3735
3d   AP:52.9174, 50.8832, 44.3638
Pedestrian AP_R40@0.50, 0.50, 0.50:
bev  AP:65.5342, 63.7425, 55.8026
3d   AP:51.2340, 49.0896, 42.5524
Pedestrian AP@0.50, 0.25, 0.25:
bev  AP:75.5410, 73.5894, 64.9255
3d   AP:75.4733, 73.3597, 64.7733
Pedestrian AP_R40@0.50, 0.25, 0.25:
bev  AP:77.0027, 74.3563, 65.3722
3d   AP:76.9066, 74.2176, 65.2428
Pickup_Truck AP@0.70, 0.70, 0.70:
bev  AP:55.6864, 43.2352, 39.6115
3d   AP:44.5032, 36.5847, 32.3596
Pickup_Truck AP_R40@0.70, 0.70, 0.70:
bev  AP:55.0906, 42.1824, 37.0217
3d   AP:42.5431, 33.1672, 28.4792
Pickup_Truck AP@0.50, 0.50, 0.50:
bev  AP:66.6935, 53.3998, 45.8237
3d   AP:66.2410, 53.2103, 45.5989
Pickup_Truck AP_R40@0.50, 0.50, 0.50:
bev  AP:66.4941, 51.9621, 45.0414
3d   AP:66.2318, 51.7528, 44.8418

Thanks a lot @mpitropov :)

mpitropov commented 2 years ago

Nice, I'll close this issue then.