About reproduce CBGS result

[muzi2045]

Instructions To Reproduce the Issue:

1. what changes you made (git diff) or what code you wrote using the Ego velocity in every annotations, the other config are same.

   # convert velo from global to lidar
   for i in range(len(ref_boxes)):
   velo = np.array([*velocity[i], 0.0])
   velo = velo @ np.linalg.inv(e2g_r_mat).T @ np.linalg.inv(
         l2e_r_mat).T
   velocity[i] = velo[:2]
   velocity = velocity.reshape(-1,2)

2. what exact command you run:

   python3 -m torch.distributed.launch --nproc_per_node=8 ./tools/train.py examples/cbgs/configs/nusc_all_vfev3_spmiddleresnetfhd_rpn2_mghead_syncbn.py --work_dir=/home/ubuntu/Documents/Det3D/trained_model

3. what you observed:

   
   mAP: 0.3719
   mATE: 0.3724
   mASE: 0.2661
   mAOE: 0.9296
   mAVE: 1.3655
   mAAE: 0.2684
   NDS: 0.4023
   Eval time: 140.1s

Per-class results: Object Class AP ATE ASE AOE AVE AAE car 0.721 0.219 0.158 0.841 1.116 0.230 truck 0.371 0.426 0.198 0.640 1.155 0.307 bus 0.500 0.439 0.174 1.223 2.171 0.431 trailer 0.213 0.687 0.219 0.670 1.371 0.184 construction_vehicle 0.058 0.798 0.481 1.370 0.157 0.372 pedestrian 0.653 0.165 0.287 1.350 0.869 0.439 motorcycle 0.242 0.223 0.243 1.107 3.192 0.153 bicycle 0.043 0.199 0.264 1.111 0.894 0.031 traffic_cone 0.449 0.170 0.348 nan nan nan barrier 0.470 0.398 0.289 0.056 nan nan Evaluation nusc: Nusc v1.0-trainval Evaluation car Nusc dist AP@0.5, 1.0, 2.0, 4.0 59.48, 71.97, 77.40, 79.65 mean AP: 0.7212472062431424 truck Nusc dist AP@0.5, 1.0, 2.0, 4.0 18.48, 36.29, 44.83, 48.91 mean AP: 0.3712787143771077 construction_vehicle Nusc dist AP@0.5, 1.0, 2.0, 4.0 0.00, 2.09, 8.06, 12.96 mean AP: 0.05777510817362395 bus Nusc dist AP@0.5, 1.0, 2.0, 4.0 23.60, 47.30, 62.94, 66.32 mean AP: 0.5003920838518946 trailer Nusc dist AP@0.5, 1.0, 2.0, 4.0 1.66, 13.24, 29.62, 40.49 mean AP: 0.21251682647224052 barrier Nusc dist AP@0.5, 1.0, 2.0, 4.0 26.14, 46.78, 55.95, 59.13 mean AP: 0.4700045657239055 motorcycle Nusc dist AP@0.5, 1.0, 2.0, 4.0 20.39, 24.74, 25.62, 26.05 mean AP: 0.24202605811125658 bicycle Nusc dist AP@0.5, 1.0, 2.0, 4.0 3.93, 4.27, 4.35, 4.58 mean AP: 0.04280152387228541 pedestrian Nusc dist AP@0.5, 1.0, 2.0, 4.0 62.12, 64.40, 66.13, 68.36 mean AP: 0.6525328516852104 traffic_cone Nusc dist AP@0.5, 1.0, 2.0, 4.0 40.81, 43.40, 45.49, 49.80 mean AP: 0.44874109465427564

![9c9f22a58fdc45f2b8a119cda3554f1f](https://user-images.githubusercontent.com/12984898/72032371-22fcd900-32ca-11ea-9fcb-dd660bfc36f0.png)
![93fdce35d7db4764ad5f822f57ab49e2](https://user-images.githubusercontent.com/12984898/72032396-35771280-32ca-11ea-9095-35046e5e99cc.png)

Unable to reproduce the results in model zoo.

## Expected behavior:
the score NDS don't reach the number in released paper, and the AVE number is abnormal large than others, this reproduced result even worse than pointpillars. Is the loss compute func exist some problems leading to this result?

[poodarchu]

check your data's velocity dim, it should be last dim of point clouds.

[poodarchu]

I advice you use my version of nuScenes, or you need to add velocity attribute to get_boxes().

[muzi2045]

check your data's velocity dim, it should be last dim of point clouds.

the velocity dim in generate reg_targets are in [x, y, z, w ,h, l, v_x, v_y, yaw] what do you mean last dim of point clouds? is the velocity info attach to per point? @poodarchu

[muzi2045]

it's the same thing, the nan velocity value are set to zero.

velocity = np.array(
     [nusc.box_velocity(token)[:2] for token in sample['anns']]
)
# convert velo from global to lidar
for i in range(len(ref_boxes)):
     velo = np.array([*velocity[i], 0.0])
     velo = velo @ np.linalg.inv(e2g_r_mat).T @ np.linalg.inv(
            l2e_r_mat).T
     velocity[i] = velo[:2]
velocity = velocity.reshape(-1,2)

gt_boxes = np.concatenate(
     [locs, dims, velocity[:, :2], -rots - np.pi / 2], axis=1
)

[muzi2045]

refer this closed issue https://github.com/poodarchu/Det3D/issues/47

[poodarchu]

I refactor the repo after the nuScenes Detection challenge, so there should be something wrong with data generation, I'll check this.

[poodarchu]

it seems that once you fix velocity targets, the results should be the same as what the paper claimed.

[muzi2045]

please keep this issue open, I am waiting for other person trained the CBGS with the fork version nuscenes-devkit, it will be result reproduced these days.

[muzi2045]

the WeightedSmoothL1Loss module, why the self._code_weights set to None?

# if code_weights is not None:
#     self._code_weights = torch.tensor(code_weights,
#                                       dtype=torch.float32)
# else:
#     self._code_weights = None
self._code_weights = None
self._codewise = codewise
self._reduction = reduction
self._loss_weight = loss_weight

[poodarchu]

the WeightedSmoothL1Loss module, why the self._code_weights set to None?

# if code_weights is not None:
#     self._code_weights = torch.tensor(code_weights,
#                                       dtype=torch.float32)
# else:
#     self._code_weights = None
self._code_weights = None
self._codewise = codewise
self._reduction = reduction
self._loss_weight = loss_weight

that's a bug, you can uncomment the corresond part. Thanks very much.

[peiyunh]

I also had a hard time reproducing the performance of CBGS on nuscenes. More can be found in #47.

[muzi2045]

fix the box_loss bug seems like the velocity will have much less influence on total loss. it will be useful to boost the mAP in eval. @peiyunh

[peiyunh]

By fixing code_weights, the loss should be affected by velocity predictions much less (0.2x as opposed to 1.0x for each direction).

I will try it out. Thanks. @muzi2045

[poodarchu]

fix the box_loss bug seems like the velocity will have much less influence on total loss. it will be useful to boost the mAP in eval. @peiyunh

Does the final result match the mAP / NDS in the paper?

[muzi2045]

after fix code_weights bug, the reproduce result: it boost a little, but it still can't reach the NDS which the paper claimed. @poodarchu

mAP: 0.4197
mATE: 0.3612
mASE: 0.2619
mAOE: 0.8838
mAVE: 1.4260
mAAE: 0.2962
NDS: 0.4296
Eval time: 136.5s

Per-class results:
Object Class    AP  ATE ASE AOE AVE AAE
car 0.745   0.207   0.155   0.822   1.208   0.250
truck   0.398   0.406   0.196   0.624   1.395   0.335
bus 0.541   0.418   0.175   0.976   2.251   0.428
trailer 0.265   0.700   0.212   0.998   1.658   0.192
construction_vehicle    0.070   0.830   0.466   1.068   0.132   0.367
pedestrian  0.693   0.156   0.287   1.387   0.954   0.653
motorcycle  0.336   0.211   0.232   1.054   2.938   0.122
bicycle 0.095   0.203   0.264   0.978   0.872   0.023
traffic_cone    0.512   0.156   0.342   nan nan nan
barrier 0.542   0.326   0.290   0.048   nan nan
Evaluation nusc: Nusc v1.0-trainval Evaluation
car Nusc dist AP@0.5, 1.0, 2.0, 4.0
62.15, 74.62, 79.50, 81.74 mean AP: 0.7450174477548169
truck Nusc dist AP@0.5, 1.0, 2.0, 4.0
20.45, 38.98, 47.61, 51.98 mean AP: 0.3975560521434358
construction_vehicle Nusc dist AP@0.5, 1.0, 2.0, 4.0
0.00, 1.77, 9.24, 17.03 mean AP: 0.07009568794018632
bus Nusc dist AP@0.5, 1.0, 2.0, 4.0
27.63, 51.69, 66.67, 70.45 mean AP: 0.5410969982901599
trailer Nusc dist AP@0.5, 1.0, 2.0, 4.0
1.51, 17.67, 35.39, 51.57 mean AP: 0.26537206743920766
barrier Nusc dist AP@0.5, 1.0, 2.0, 4.0
36.26, 54.43, 61.72, 64.39 mean AP: 0.5420002219625686
motorcycle Nusc dist AP@0.5, 1.0, 2.0, 4.0
29.04, 34.27, 35.32, 35.86 mean AP: 0.33621267948326206
bicycle Nusc dist AP@0.5, 1.0, 2.0, 4.0
8.75, 9.57, 9.72, 10.08 mean AP: 0.09530030931634574
pedestrian Nusc dist AP@0.5, 1.0, 2.0, 4.0
66.35, 68.35, 70.08, 72.38 mean AP: 0.6929089661819542
traffic_cone Nusc dist AP@0.5, 1.0, 2.0, 4.0
47.69, 49.81, 51.59, 55.66 mean AP: 0.5118637198071521

[poodarchu]

Nor the problem should be with AOE, you can check direction classifier.

[muzi2045]

maybe I can try not use direction classifier and set the velocity code weight to zero. do you have any loss logs about the training process? I want to know which part get problem.

[poodarchu]

You can check the nuscenes leaderboard, and compare current metrics with ones on the leaderboard. The AVE should be about 0.3.

[peiyunh]

Here are my results after fixing the code_weights (@muzi2045 ):

Saving metrics to: /home/peiyunh/work/Det3D/models/NUSC_CBGS_reproduce_20200109_231011
mAP: 0.4297
mATE: 0.3581
mASE: 0.2614
mAOE: 0.8661
mAVE: 1.3247
mAAE: 0.2568
NDS: 0.4406
Eval time: 75.8s
2020-01-11 02:21:01,974 - INFO -

2020-01-11 02:21:01,977 - INFO - Evaluation nusc: Nusc v1.0-trainval Evaluation
car Nusc dist AP@0.5, 1.0, 2.0, 4.0
62.28, 74.15, 79.40, 81.69 mean AP: 0.7437893086443743
truck Nusc dist AP@0.5, 1.0, 2.0, 4.0
19.09, 37.96, 46.76, 51.74 mean AP: 0.388894998372028
construction_vehicle Nusc dist AP@0.5, 1.0, 2.0, 4.0
0.00, 2.64, 10.42, 16.95 mean AP: 0.0750089933054364
bus Nusc dist AP@0.5, 1.0, 2.0, 4.0
27.83, 54.44, 67.62, 71.27 mean AP: 0.5528683031823456
trailer Nusc dist AP@0.5, 1.0, 2.0, 4.0
1.51, 17.91, 38.79, 55.37 mean AP: 0.28395019718021886
barrier Nusc dist AP@0.5, 1.0, 2.0, 4.0
36.49, 54.62, 61.90, 64.53 mean AP: 0.5438601444345098
motorcycle Nusc dist AP@0.5, 1.0, 2.0, 4.0
30.10, 36.34, 37.91, 38.56 mean AP: 0.3572533407633828
bicycle Nusc dist AP@0.5, 1.0, 2.0, 4.0
12.84, 14.12, 14.50, 14.93 mean AP: 0.14094316234276158
pedestrian Nusc dist AP@0.5, 1.0, 2.0, 4.0
66.79, 68.38, 69.97, 72.15 mean AP: 0.6932140499900883
traffic_cone Nusc dist AP@0.5, 1.0, 2.0, 4.0
47.99, 49.96, 52.35, 56.60 mean AP: 0.5172578714562024

My velocity-based detection error (AVE) is still much larger than what's reported and the mAP is much lower. Were you able to reproduce the reported numbers using the released code? @poodarchu

[poodarchu]

I will check this, I haven't run the models after I refactor the codebase.

[muzi2045]

The provided checkpoint is a single model with 51.91% mAP and 62.83% NDS.
Its input grid size is (0.05m, 0.05m, 0.2m). 
The cell below shows result details on Val Split.

the setting in config.py is

voxel_generator = dict(
    range=[-50.4, -50.4, -5.0, 50.4, 50.4, 3.0],
    voxel_size=[0.1, 0.1, 0.2],
    max_points_in_voxel=10,
    max_voxel_num=60000,
)

@poodarchu

[peiyunh]

I will check this, I haven't run the models after I refactor the codebase.

Thanks!

The provided checkpoint is a single model with 51.91% mAP and 62.83% NDS.
Its input grid size is (0.05m, 0.05m, 0.2m). 
The cell below shows result details on Val Split.

the setting in config.py is

voxel_generator = dict(
    range=[-50.4, -50.4, -5.0, 50.4, 50.4, 3.0],
    voxel_size=[0.1, 0.1, 0.2],
    max_points_in_voxel=10,
    max_voxel_num=60000,
)

@muzi2045 Where did you see an input grid size of [0.05m, 0.05m, 0.2m] mentioned? Have you tried training with this grid size? It seems GTX 2080 Ti GPUs do not offer enough memory to do so.

[muzi2045]

it's mentioned in ./examples/cbgs/README.md I am training with this config , per GPU with one sample, But it will much slower than previous config( 5 days for 20 epochs training). @peiyunh

[peiyunh]

@muzi2045 Have you tried enabling the ground plane during training? The tech report mentioned it was used but in the create_data.py it was initially commented out.

[muzi2045]

the ground plane is not enabled, and I can't find it where used this func.

[poodarchu]

the ground plane is not enabled, and I can't find it where used this func.

It doesn't need to use ground plane detection module to reach 52.8 mAP

[muzi2045]

what about the reproduce result with this code base in your training test？

[turboxin]

hi @muzi2045 @peiyunh, have you tried to train pointpillars on nuscene? And if you do, could you please report the eval result? Thank you a lot~

[peiyunh]

@turboxin I have not tried this repo for PointPillars. I have used SECOND's codebase (https://github.com/traveller59/second.pytorch) for PointPillars. Numbers can be found in Table 2 of this recent tech report: https://arxiv.org/abs/1912.04986. The reproduced mAP is about 2% higher than the official PointPillars on NuScenes's validation set.

[muzi2045]

I noticed the loss compute are extremmly unstable in the training process. @peiyunh have you noticed this shake in loss print value?

[MeyLavie]

Hi, I'm using @poodarchu Nuscenes version and still get these results:

mAP: 0.3835
mATE: 0.3727
mASE: 0.2656
mAOE: 0.8735
mAVE: 1.3856
mAAE: 0.2581
NDS: 0.4148
Eval time: 89.0s
2020-01-14 21:14:07,617 - INFO - 

2020-01-14 21:14:07,619 - INFO - Evaluation nusc: Nusc v1.0-trainval Evaluation
car Nusc dist AP@0.5, 1.0, 2.0, 4.0
59.47, 72.11, 77.42, 79.73 mean AP: 0.7218301569237199
truck Nusc dist AP@0.5, 1.0, 2.0, 4.0
18.09, 36.22, 45.09, 49.56 mean AP: 0.372367428233392
construction_vehicle Nusc dist AP@0.5, 1.0, 2.0, 4.0
0.00, 2.45, 8.28, 13.50 mean AP: 0.060575765814401564
bus Nusc dist AP@0.5, 1.0, 2.0, 4.0
23.29, 47.85, 63.37, 67.68 mean AP: 0.5054781662286532
trailer Nusc dist AP@0.5, 1.0, 2.0, 4.0
1.61, 14.39, 30.38, 44.41 mean AP: 0.2269405262726112
barrier Nusc dist AP@0.5, 1.0, 2.0, 4.0
24.70, 47.08, 56.79, 59.97 mean AP: 0.4713448151395909
motorcycle Nusc dist AP@0.5, 1.0, 2.0, 4.0
23.87, 27.94, 28.82, 29.51 mean AP: 0.27537349210694445
bicycle Nusc dist AP@0.5, 1.0, 2.0, 4.0
7.67, 8.49, 9.13, 9.47 mean AP: 0.08691368416913861
pedestrian Nusc dist AP@0.5, 1.0, 2.0, 4.0
62.66, 64.41, 66.30, 68.26 mean AP: 0.6540481323410499
traffic_cone Nusc dist AP@0.5, 1.0, 2.0, 4.0
42.31, 44.48, 46.41, 50.81 mean AP: 0.46001472906626706

[poodarchu]

I'll check this during Chinese New Year

[HenryJunW]

@peiyunh For the second github repository, which config file were you using for reproducing the results you got? If possible, which version of Second were you using? Thanks.

[countriccati]

Has anyone been able to reproduce the published results training on nuscenes? @qchenclaire you seem to be the only one coming close, can you explain what changes you made? https://github.com/poodarchu/Det3D/issues/47#issuecomment-574800408

@poodarchu have you had a chance to look into this?

[turboxin]

Hi @poodarchu, according to your paper, you ensembled severval models of multiple scales for final submission. Could you please kindly explain what multiple scale means, does it mean multiple voxel size?

[poodarchu]

Hi @poodarchu, according to your paper, you ensembled severval models of multiple scales for final submission. Could you please kindly explain what multiple scale means, does it mean multiple voxel size?

Yes

[Linkon87]

@muzi2045 Hi! I have another question,I notice that you successfully got the detection results of lidar pictures,could you please kindly tell me how to realize it? which file & commands I should use?