improve baselines Closes #55, Closes #47, Closes #80, Closes #117

tianweiy commented 4 years ago

This pr is trying to fix #80 #117 #55 #47.

I get an improved baselines based on a forked version of det3d. And now I move some of the changes from my repo to the original one.

PointPillars:

Performance Improvement:

before this commit:

mAP: 0.2992                                                                                                                                                                                                                                   
mATE: 0.4408                                                                                                                                                                                                                                  
mASE: 0.2704                                                                                                                                                                                                                                  
mAOE: 0.9446                                                                                                                                                                                                                                  
mAVE: 0.5465                                                                                                                                                                                                                                  
mAAE: 0.2187                                                                                                                                                                                                                                  
NDS: 0.4075                                                                                                                                                                                                                                   
Eval time: 194.4s                                                                                                                                                                                                                             

Per-class results:                                                                                                                                                                                                                            
Object Class    AP      ATE     ASE     AOE     AVE     AAE                                                                                                                                                                                   
car     0.732   0.236   0.161   0.788   0.261   0.213                                                                                                                                                                                         
truck   0.336   0.470   0.210   0.702   0.298   0.249                                                                                                                                                                                         
bus     0.395   0.518   0.188   0.953   0.851   0.266                                                                                                                                                                                         
trailer 0.197   0.697   0.221   0.698   0.222   0.183                                                                                                                                                                                         
construction_vehicle    0.038   0.802   0.467   1.328   0.138   0.328                                                                                                                                                                         
pedestrian      0.543   0.328   0.282   1.450   1.013   0.213                                                                                                                                                                                 
motorcycle      0.109   0.292   0.237   1.477   0.983   0.274
bicycle 0.005   0.290   0.265   1.043   0.605   0.023
traffic_cone    0.265   0.273   0.365   nan     nan     nan
barrier 0.372   0.503   0.308   0.063   nan     nan

After this commit:

mAP: 0.4179
mATE: 0.3630
mASE: 0.2636
mAOE: 0.3770
mAVE: 0.2877
mAAE: 0.1983
NDS: 0.5600
Eval time: 81.2s

Per-class results:
Object Class    AP      ATE     ASE     AOE     AVE     AAE
car     0.801   0.215   0.159   0.191   0.251   0.205
truck   0.457   0.411   0.205   0.159   0.192   0.223
bus     0.558   0.444   0.192   0.126   0.532   0.308
trailer 0.278   0.617   0.202   0.478   0.157   0.172
construction_vehicle    0.103   0.757   0.434   1.159   0.114   0.346
pedestrian      0.717   0.163   0.277   0.432   0.244   0.088
motorcycle      0.321   0.230   0.246   0.354   0.599   0.215
bicycle 0.053   0.193   0.265   0.376   0.213   0.030
traffic_cone    0.437   0.188   0.358   nan     nan     nan
barrier 0.455   0.411   0.297   0.117   nan     nan
Evaluation nusc: Nusc v1.0-trainval Evaluation
car Nusc dist AP@0.5, 1.0, 2.0, 4.0
67.75, 81.12, 85.15, 86.46 mean AP: 0.8012047920593498
truck Nusc dist AP@0.5, 1.0, 2.0, 4.0
23.71, 45.39, 54.92, 58.94 mean AP: 0.45739047966566226
construction_vehicle Nusc dist AP@0.5, 1.0, 2.0, 4.0
0.14, 5.21, 15.55, 20.37 mean AP: 0.10317209147626494
bus Nusc dist AP@0.5, 1.0, 2.0, 4.0
26.23, 54.24, 69.76, 72.88 mean AP: 0.5577483330153783
trailer Nusc dist AP@0.5, 1.0, 2.0, 4.0
5.09, 21.19, 35.53, 49.41 mean AP: 0.2780347493252949
barrier Nusc dist AP@0.5, 1.0, 2.0, 4.0
25.15, 45.56, 53.82, 57.33 mean AP: 0.45465027132389
motorcycle Nusc dist AP@0.5, 1.0, 2.0, 4.0
27.79, 32.90, 33.57, 33.96 mean AP: 0.3205635980429202
bicycle Nusc dist AP@0.5, 1.0, 2.0, 4.0
5.04, 5.26, 5.28, 5.45 mean AP: 0.05258777460814078
pedestrian Nusc dist AP@0.5, 1.0, 2.0, 4.0
68.81, 70.60, 72.70, 74.61 mean AP: 0.7168225670676128
traffic_cone Nusc dist AP@0.5, 1.0, 2.0, 4.0
39.71, 41.46, 44.16, 49.28 mean AP: 0.4365392833724403

Summary of changes

bug fixes in pillar encoder (this boosts to 39.1 map / 50.2 nds)
cos / sin encoding in contrast to directional encoder (both the original repo and my current one can't get directional classifier work on nuScenes) -> (35.4 map / 52.4 nds)
flip along x/y axis during training + regression weight from 1 to 0.25 + filtering zero points box during training -> (41.8 map / 56 nds)

In my repo(which will come out next Monday), I can get (45.5 map / 58.4 nds). The remained changes to my knowledge are:

I use a heavier head with 3 conv-bn-relu. (this should matter the most)
I use l1 loss in place of smooth-l1 loss (minor accuracy difference)
I initialize the final classification conv layer's bias to -2.19 (following focal paper, not sure how much improvement)

Changing to a heavier head requires rewriting mg_head so I didn't do this here.

CBGS

results will be added once finish.(tomorrow night)

To use this newest commit, you need to regenerate all those info files to filter out zero point boxes during training. The other should be the same (no need to regenerate the gt database if you already have one).

I will provide pre-trained models once the pr is merged.

poodarchu commented 4 years ago

Can you provide corresponding checkpoints and training logs?

tianweiy commented 4 years ago

yes, the pointpillars log and checkpoint are available here. I will add cbgs in 12 hours.

tianweiy commented 4 years ago

CBGS Updates:

before:

46.7 / 54.55 (copied from issue section)

mAP: 0.4669
mATE: 0.3391
mASE: 0.2574
mAOE: 0.7657
mAVE: 0.3162
mAAE: 0.2012
NDS: 0.5455
Eval time: 89.3s
2020-01-18 08:06:51,504 - INFO - 

2020-01-18 08:06:51,505 - INFO - Evaluation nusc: Nusc v1.0-trainval Evaluation
car Nusc dist AP@0.5, 1.0, 2.0, 4.0
68.54, 80.63, 84.63, 86.58 mean AP: 0.8009424366048317
truck Nusc dist AP@0.5, 1.0, 2.0, 4.0
24.87, 44.61, 55.15, 58.78 mean AP: 0.45850578704319195
construction_vehicle Nusc dist AP@0.5, 1.0, 2.0, 4.0
0.17, 6.90, 16.21, 24.29 mean AP: 0.11890247733120854
bus Nusc dist AP@0.5, 1.0, 2.0, 4.0
32.94, 56.79, 73.28, 76.13 mean AP: 0.5978476967252525
trailer Nusc dist AP@0.5, 1.0, 2.0, 4.0
4.21, 20.78, 37.94, 53.70 mean AP: 0.29155996509713616
barrier Nusc dist AP@0.5, 1.0, 2.0, 4.0
39.77, 55.13, 59.95, 62.47 mean AP: 0.5433129255795658
motorcycle Nusc dist AP@0.5, 1.0, 2.0, 4.0
31.97, 39.27, 40.01, 40.37 mean AP: 0.3790460790422441
bicycle Nusc dist AP@0.5, 1.0, 2.0, 4.0
13.72, 14.36, 14.47, 14.65 mean AP: 0.1430066611617972
pedestrian Nusc dist AP@0.5, 1.0, 2.0, 4.0
72.81, 75.17, 76.99, 78.74 mean AP: 0.7592805073941438
traffic_cone Nusc dist AP@0.5, 1.0, 2.0, 4.0
53.54, 55.51, 58.53, 63.15 mean AP: 0.5768369346063308

now:

mAP: 0.4990
mATE: 0.3353
mASE: 0.2563
mAOE: 0.3230
mAVE: 0.2505
mAAE: 0.1969
NDS: 0.6133
Eval time: 105.2s

Per-class results:
Object Class    AP  ATE ASE AOE AVE AAE
car 0.818   0.195   0.154   0.111   0.249   0.209
truck   0.492   0.392   0.197   0.097   0.199   0.229
bus 0.629   0.392   0.185   0.061   0.394   0.246
trailer 0.316   0.633   0.199   0.412   0.154   0.163
construction_vehicle    0.137   0.783   0.449   1.144   0.125   0.339
pedestrian  0.784   0.161   0.280   0.397   0.220   0.093
motorcycle  0.444   0.202   0.234   0.278   0.479   0.281
bicycle 0.208   0.166   0.263   0.319   0.185   0.015
traffic_cone    0.589   0.152   0.326   nan nan nan
barrier 0.574   0.277   0.276   0.089   nan nan
Evaluation nusc: Nusc v1.0-trainval Evaluation
car Nusc dist AP@0.5, 1.0, 2.0, 4.0
70.72, 82.64, 86.00, 87.89 mean AP: 0.8181075796960271
truck Nusc dist AP@0.5, 1.0, 2.0, 4.0
27.03, 49.02, 58.66, 62.18 mean AP: 0.4922299142139578
construction_vehicle Nusc dist AP@0.5, 1.0, 2.0, 4.0
0.30, 6.93, 19.51, 27.89 mean AP: 0.13656648365947183
bus Nusc dist AP@0.5, 1.0, 2.0, 4.0
35.38, 60.28, 76.32, 79.46 mean AP: 0.6285951454369655
trailer Nusc dist AP@0.5, 1.0, 2.0, 4.0
4.82, 25.11, 43.58, 52.77 mean AP: 0.315690082891094
barrier Nusc dist AP@0.5, 1.0, 2.0, 4.0
43.22, 57.75, 62.99, 65.45 mean AP: 0.5735325513729088
motorcycle Nusc dist AP@0.5, 1.0, 2.0, 4.0
39.19, 45.68, 46.28, 46.57 mean AP: 0.4442935931154901
bicycle Nusc dist AP@0.5, 1.0, 2.0, 4.0
20.15, 20.96, 21.02, 21.11 mean AP: 0.2081029978740392
pedestrian Nusc dist AP@0.5, 1.0, 2.0, 4.0
75.36, 77.71, 79.50, 81.18 mean AP: 0.7843566320006093
traffic_cone Nusc dist AP@0.5, 1.0, 2.0, 4.0
55.22, 57.02, 59.53, 63.66 mean AP: 0.5885651075883424

all files are available here

poodarchu commented 4 years ago

can you update the results in readme with links?

tianweiy commented 4 years ago

sure.

tianweiy commented 4 years ago

done

V2AI / Det3D

improve baselines Closes #55, Closes #47, Closes #80, Closes #117 #119