search

dvlab-research / UVTR

Unifying Voxel-based Representation with Transformer for 3D Object Detection (NeurIPS 2022)
227 stars 17 forks source link

low results on nusenes-mini #26

Open zizo1111 opened 1 year ago

zizo1111 commented 1 year ago

Hi, I was trying to evaluate the pretrained models on the nuscenes mini data, but I am getting very bad results which is unexpected. I have mmdet3d v1.0.0.rc4 installed

Do you maybe know what the problem is?

Thanks :)

here is the log

user@gpu-cluster:~/src/mmdet$ python3 extra_tools/test.py configs/uvtr/multi_modality/uvtr_m_v0075_r101_h5.py models/uvtr/uvtr_m_v0075_r101_h5.pth --eval=bbox --out mini_test.pkl
/home/user/.local/lib/python3.8/site-packages/mmcv/__init__.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.
  warnings.warn(
/home/user/.local/lib/python3.8/site-packages/mmdet/utils/setup_env.py:38: UserWarning: Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
  warnings.warn(
/home/user/.local/lib/python3.8/site-packages/mmdet/utils/setup_env.py:48: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
  warnings.warn(
/usr/local/lib/python3.8/dist-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
load checkpoint from local path: models/uvtr/uvtr_m_v0075_r101_h5.pth
2023-04-25 16:42:17,693 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.0.conv2 is upgraded to version 2.
2023-04-25 16:42:17,695 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.1.conv2 is upgraded to version 2.
2023-04-25 16:42:17,697 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.2.conv2 is upgraded to version 2.
2023-04-25 16:42:17,699 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.3.conv2 is upgraded to version 2.
2023-04-25 16:42:17,701 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.4.conv2 is upgraded to version 2.
2023-04-25 16:42:17,703 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.5.conv2 is upgraded to version 2.
2023-04-25 16:42:17,704 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.6.conv2 is upgraded to version 2.
2023-04-25 16:42:17,706 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.7.conv2 is upgraded to version 2.
2023-04-25 16:42:17,708 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.8.conv2 is upgraded to version 2.
2023-04-25 16:42:17,710 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.9.conv2 is upgraded to version 2.
2023-04-25 16:42:17,712 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.10.conv2 is upgraded to version 2.
2023-04-25 16:42:17,714 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.11.conv2 is upgraded to version 2.
2023-04-25 16:42:17,716 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.12.conv2 is upgraded to version 2.
2023-04-25 16:42:17,717 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.13.conv2 is upgraded to version 2.
2023-04-25 16:42:17,719 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.14.conv2 is upgraded to version 2.
2023-04-25 16:42:17,721 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.15.conv2 is upgraded to version 2.
2023-04-25 16:42:17,723 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.16.conv2 is upgraded to version 2.
2023-04-25 16:42:17,725 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.17.conv2 is upgraded to version 2.
2023-04-25 16:42:17,728 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.18.conv2 is upgraded to version 2.
2023-04-25 16:42:17,730 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.19.conv2 is upgraded to version 2.
2023-04-25 16:42:17,732 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.20.conv2 is upgraded to version 2.
2023-04-25 16:42:17,733 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.21.conv2 is upgraded to version 2.
2023-04-25 16:42:17,735 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.22.conv2 is upgraded to version 2.
2023-04-25 16:42:17,737 - root - INFO - ModulatedDeformConvPack img_backbone.layer4.0.conv2 is upgraded to version 2.
2023-04-25 16:42:17,741 - root - INFO - ModulatedDeformConvPack img_backbone.layer4.1.conv2 is upgraded to version 2.
2023-04-25 16:42:17,744 - root - INFO - ModulatedDeformConvPack img_backbone.layer4.2.conv2 is upgraded to version 2.
[                                                  ] 0/81, elapsed: 0s, ETA:/usr/local/lib/python3.8/dist-packages/torch/nn/functional.py:4227: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
  warnings.warn(
/src/mmdet3d/core/bbox/coders/nms_free_coder.py:67: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  self.post_center_range = torch.tensor(
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 81/81, 2.6 task/s, elapsed: 32s, ETA:     0s
writing results to mini_test.pkl

Formating bboxes of pts_bbox
Start to convert detection format...
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 81/81, 23.1 task/s, elapsed: 3s, ETA:     0s
Results writes to /tmp/tmps9zh56ms/results/pts_bbox/results_nusc.json
Evaluating bboxes of pts_bbox
mAP: 0.1188                                                                                                                                             
mATE: 0.8707
mASE: 0.5113
mAOE: 0.6999
mAVE: 1.0449
mAAE: 0.4827
NDS: 0.2029
Eval time: 4.8s

Per-class results:
Object Class    AP      ATE     ASE     AOE     AVE     AAE
car     0.204   0.872   0.186   0.334   0.212   0.065
truck   0.075   0.673   0.290   0.355   0.216   0.118
bus     0.017   1.250   0.216   0.406   3.521   1.000
trailer 0.000   1.000   1.000   1.000   1.000   1.000
construction_vehicle    0.000   1.000   1.000   1.000   1.000   1.000
pedestrian      0.319   0.582   0.308   0.673   0.572   0.367
motorcycle      0.212   0.869   0.317   0.922   0.070   0.011
bicycle 0.095   0.636   0.372   0.609   1.767   0.301
traffic_cone    0.265   0.825   0.425   nan     nan     nan
barrier 0.000   1.000   1.000   1.000   nan     nan
{'pts_bbox_NuScenes/car_AP_dist_0.5': 0.006, 'pts_bbox_NuScenes/car_AP_dist_1.0': 0.1039, 'pts_bbox_NuScenes/car_AP_dist_2.0': 0.2847, 'pts_bbox_NuScenes/car_AP_dist_4.0': 0.4234, 'pts_bbox_NuScenes/car_trans_err': 0.8717, 'pts_bbox_NuScenes/car_scale_err': 0.1856, 'pts_bbox_NuScenes/car_orient_err': 0.3339, 'pts_bbox_NuScenes/car_vel_err': 0.2123, 'pts_bbox_NuScenes/car_attr_err': 0.0646, 'pts_bbox_NuScenes/mATE': 0.8707, 'pts_bbox_NuScenes/mASE': 0.5113, 'pts_bbox_NuScenes/mAOE': 0.6999, 'pts_bbox_NuScenes/mAVE': 1.0449, 'pts_bbox_NuScenes/mAAE': 0.4827, 'pts_bbox_NuScenes/truck_AP_dist_0.5': 0.0037, 'pts_bbox_NuScenes/truck_AP_dist_1.0': 0.0371, 'pts_bbox_NuScenes/truck_AP_dist_2.0': 0.0762, 'pts_bbox_NuScenes/truck_AP_dist_4.0': 0.1825, 'pts_bbox_NuScenes/truck_trans_err': 0.673, 'pts_bbox_NuScenes/truck_scale_err': 0.2899, 'pts_bbox_NuScenes/truck_orient_err': 0.3548, 'pts_bbox_NuScenes/truck_vel_err': 0.2161, 'pts_bbox_NuScenes/truck_attr_err': 0.118, 'pts_bbox_NuScenes/construction_vehicle_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/construction_vehicle_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/construction_vehicle_AP_dist_2.0': 0.0, 'pts_bbox_NuScenes/construction_vehicle_AP_dist_4.0': 0.0, 'pts_bbox_NuScenes/construction_vehicle_trans_err': 1.0, 'pts_bbox_NuScenes/construction_vehicle_scale_err': 1.0, 'pts_bbox_NuScenes/construction_vehicle_orient_err': 1.0, 'pts_bbox_NuScenes/construction_vehicle_vel_err': 1.0, 'pts_bbox_NuScenes/construction_vehicle_attr_err': 1.0, 'pts_bbox_NuScenes/bus_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/bus_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/bus_AP_dist_2.0': 0.0003, 'pts_bbox_NuScenes/bus_AP_dist_4.0': 0.0676, 'pts_bbox_NuScenes/bus_trans_err': 1.2499, 'pts_bbox_NuScenes/bus_scale_err': 0.216, 'pts_bbox_NuScenes/bus_orient_err': 0.4058, 'pts_bbox_NuScenes/bus_vel_err': 3.5214, 'pts_bbox_NuScenes/bus_attr_err': 1.0, 'pts_bbox_NuScenes/trailer_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/trailer_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/trailer_AP_dist_2.0': 0.0, 'pts_bbox_NuScenes/trailer_AP_dist_4.0': 0.0, 'pts_bbox_NuScenes/trailer_trans_err': 1.0, 'pts_bbox_NuScenes/trailer_scale_err': 1.0, 'pts_bbox_NuScenes/trailer_orient_err': 1.0, 'pts_bbox_NuScenes/trailer_vel_err': 1.0, 'pts_bbox_NuScenes/trailer_attr_err': 1.0, 'pts_bbox_NuScenes/barrier_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/barrier_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/barrier_AP_dist_2.0': 0.0, 'pts_bbox_NuScenes/barrier_AP_dist_4.0': 0.0, 'pts_bbox_NuScenes/barrier_trans_err': 1.0, 'pts_bbox_NuScenes/barrier_scale_err': 1.0, 'pts_bbox_NuScenes/barrier_orient_err': 1.0, 'pts_bbox_NuScenes/barrier_vel_err': nan, 'pts_bbox_NuScenes/barrier_attr_err': nan, 'pts_bbox_NuScenes/motorcycle_AP_dist_0.5': 0.0019, 'pts_bbox_NuScenes/motorcycle_AP_dist_1.0': 0.1057, 'pts_bbox_NuScenes/motorcycle_AP_dist_2.0': 0.2772, 'pts_bbox_NuScenes/motorcycle_AP_dist_4.0': 0.4626, 'pts_bbox_NuScenes/motorcycle_trans_err': 0.8689, 'pts_bbox_NuScenes/motorcycle_scale_err': 0.3167, 'pts_bbox_NuScenes/motorcycle_orient_err': 0.9223, 'pts_bbox_NuScenes/motorcycle_vel_err': 0.0699, 'pts_bbox_NuScenes/motorcycle_attr_err': 0.0111, 'pts_bbox_NuScenes/bicycle_AP_dist_0.5': 0.0086, 'pts_bbox_NuScenes/bicycle_AP_dist_1.0': 0.0796, 'pts_bbox_NuScenes/bicycle_AP_dist_2.0': 0.1092, 'pts_bbox_NuScenes/bicycle_AP_dist_4.0': 0.1837, 'pts_bbox_NuScenes/bicycle_trans_err': 0.636, 'pts_bbox_NuScenes/bicycle_scale_err': 0.3723, 'pts_bbox_NuScenes/bicycle_orient_err': 0.6093, 'pts_bbox_NuScenes/bicycle_vel_err': 1.7671, 'pts_bbox_NuScenes/bicycle_attr_err': 0.3007, 'pts_bbox_NuScenes/pedestrian_AP_dist_0.5': 0.1008, 'pts_bbox_NuScenes/pedestrian_AP_dist_1.0': 0.2702, 'pts_bbox_NuScenes/pedestrian_AP_dist_2.0': 0.3986, 'pts_bbox_NuScenes/pedestrian_AP_dist_4.0': 0.5082, 'pts_bbox_NuScenes/pedestrian_trans_err': 0.5822, 'pts_bbox_NuScenes/pedestrian_scale_err': 0.3078, 'pts_bbox_NuScenes/pedestrian_orient_err': 0.6728, 'pts_bbox_NuScenes/pedestrian_vel_err': 0.572, 'pts_bbox_NuScenes/pedestrian_attr_err': 0.3673, 'pts_bbox_NuScenes/traffic_cone_AP_dist_0.5': 0.0334, 'pts_bbox_NuScenes/traffic_cone_AP_dist_1.0': 0.2129, 'pts_bbox_NuScenes/traffic_cone_AP_dist_2.0': 0.3517, 'pts_bbox_NuScenes/traffic_cone_AP_dist_4.0': 0.4602, 'pts_bbox_NuScenes/traffic_cone_trans_err': 0.8253, 'pts_bbox_NuScenes/traffic_cone_scale_err': 0.4245, 'pts_bbox_NuScenes/traffic_cone_orient_err': nan, 'pts_bbox_NuScenes/traffic_cone_vel_err': nan, 'pts_bbox_NuScenes/traffic_cone_attr_err': nan, 'pts_bbox_NuScenes/NDS': 0.20291704194649415, 'pts_bbox_NuScenes/mAP': 0.1187504058313846}
yanwei-li commented 1 year ago

Hi, I never tested the result on nuscenes mini data. Could you reproduce the results on the nuscenes val set?

zizo1111 commented 1 year ago

Hi, I managed to free up enough space to download the whole dataset and I got similar results.

user@gpu-cluster:~/src/mmdet$ python3 extra_tools/test.py configs/uvtr/multi_modality/uvtr_m_v0075_r101_h5.py ckpts/uvtr/uvtr_m_v0075_r101_h5.pth --eval=bbox
/usr/local/lib/python3.8/dist-packages/mmcv/__init__.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.
  warnings.warn(
/usr/local/lib/python3.8/dist-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
load checkpoint from local path: ckpts/uvtr/uvtr_m_v0075_r101_h5.pth
2023-04-30 12:26:43,335 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.0.conv2 is upgraded to version 2.
2023-04-30 12:26:43,337 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.1.conv2 is upgraded to version 2.
2023-04-30 12:26:43,338 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.2.conv2 is upgraded to version 2.
2023-04-30 12:26:43,339 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.3.conv2 is upgraded to version 2.
2023-04-30 12:26:43,340 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.4.conv2 is upgraded to version 2.
2023-04-30 12:26:43,341 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.5.conv2 is upgraded to version 2.
2023-04-30 12:26:43,342 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.6.conv2 is upgraded to version 2.
2023-04-30 12:26:43,343 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.7.conv2 is upgraded to version 2.
2023-04-30 12:26:43,344 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.8.conv2 is upgraded to version 2.
2023-04-30 12:26:43,345 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.9.conv2 is upgraded to version 2.
2023-04-30 12:26:43,346 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.10.conv2 is upgraded to version 2.
2023-04-30 12:26:43,347 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.11.conv2 is upgraded to version 2.
2023-04-30 12:26:43,349 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.12.conv2 is upgraded to version 2.
2023-04-30 12:26:43,350 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.13.conv2 is upgraded to version 2.
2023-04-30 12:26:43,351 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.14.conv2 is upgraded to version 2.
2023-04-30 12:26:43,352 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.15.conv2 is upgraded to version 2.
2023-04-30 12:26:43,353 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.16.conv2 is upgraded to version 2.
2023-04-30 12:26:43,354 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.17.conv2 is upgraded to version 2.
2023-04-30 12:26:43,355 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.18.conv2 is upgraded to version 2.
2023-04-30 12:26:43,356 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.19.conv2 is upgraded to version 2.
2023-04-30 12:26:43,357 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.20.conv2 is upgraded to version 2.
2023-04-30 12:26:43,358 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.21.conv2 is upgraded to version 2.
2023-04-30 12:26:43,359 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.22.conv2 is upgraded to version 2.
2023-04-30 12:26:43,360 - root - INFO - ModulatedDeformConvPack img_backbone.layer4.0.conv2 is upgraded to version 2.
2023-04-30 12:26:43,362 - root - INFO - ModulatedDeformConvPack img_backbone.layer4.1.conv2 is upgraded to version 2.
2023-04-30 12:26:43,364 - root - INFO - ModulatedDeformConvPack img_backbone.layer4.2.conv2 is upgraded to version 2.
[                                                  ] 0/6019, elapsed: 0s, ETA:/usr/local/lib/python3.8/dist-packages/torch/nn/functional.py:4227: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
  warnings.warn(
[                                                  ] 1/6019, 0.3 task/s, elapsed: 3s, ETA: 18116s ~/src/mmdet/mmdet3d/core/bbox/coders/nms_free_coder.py:67: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  self.post_center_range = torch.tensor(
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 6019/6019, 2.1 task/s, elapsed: 2800s, ETA:     0s
Formating bboxes of pts_bbox
Start to convert detection format...
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 6019/6019, 40.5 task/s, elapsed: 149s, ETA:     0s
Results writes to /tmp/tmp2zhts1f2/results/pts_bbox/results_nusc.json
Evaluating bboxes of pts_bbox
mAP: 0.1348                                                                                               
mATE: 0.8543
mASE: 0.3209
mAOE: 0.6170
mAVE: 0.8633
mAAE: 0.2407
NDS: 0.2778
Eval time: 83.0s

Per-class results:
Object Class    AP      ATE     ASE     AOE     AVE     AAE
car     0.142   0.875   0.180   0.249   0.971   0.183
truck   0.049   0.992   0.265   0.324   0.876   0.229
bus     0.017   1.112   0.248   0.555   2.468   0.604
trailer 0.021   1.100   0.283   0.908   0.330   0.135
construction_vehicle    0.002   1.001   0.529   1.144   0.130   0.228
pedestrian      0.295   0.541   0.323   0.683   0.649   0.316
motorcycle      0.177   0.681   0.296   0.629   1.075   0.194
bicycle 0.253   0.663   0.311   0.703   0.408   0.038
traffic_cone    0.225   0.773   0.394   nan     nan     nan
barrier 0.166   0.803   0.380   0.359   nan     nan
{'pts_bbox_NuScenes/car_AP_dist_0.5': 0.0019, 'pts_bbox_NuScenes/car_AP_dist_1.0': 0.0591, 'pts_bbox_NuScenes/car_AP_dist_2.0': 0.191, 'pts_bbox_NuScenes/car_AP_dist_4.0': 0.316, 'pts_bbox_NuScenes/car_trans_err': 0.8745, 'pts_bbox_NuScenes/car_scale_err': 0.1798, 'pts_bbox_NuScenes/car_orient_err': 0.2486, 'pts_bbox_NuScenes/car_vel_err': 0.9711, 'pts_bbox_NuScenes/car_attr_err': 0.1828, 'pts_bbox_NuScenes/mATE': 0.8543, 'pts_bbox_NuScenes/mASE': 0.3209, 'pts_bbox_NuScenes/mAOE': 0.617, 'pts_bbox_NuScenes/mAVE': 0.8633, 'pts_bbox_NuScenes/mAAE': 0.2407, 'pts_bbox_NuScenes/truck_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/truck_AP_dist_1.0': 0.0057, 'pts_bbox_NuScenes/truck_AP_dist_2.0': 0.0585, 'pts_bbox_NuScenes/truck_AP_dist_4.0': 0.1328, 'pts_bbox_NuScenes/truck_trans_err': 0.9924, 'pts_bbox_NuScenes/truck_scale_err': 0.2653, 'pts_bbox_NuScenes/truck_orient_err': 0.3236, 'pts_bbox_NuScenes/truck_vel_err': 0.8758, 'pts_bbox_NuScenes/truck_attr_err': 0.2292, 'pts_bbox_NuScenes/construction_vehicle_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/construction_vehicle_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/construction_vehicle_AP_dist_2.0': 0.0002, 'pts_bbox_NuScenes/construction_vehicle_AP_dist_4.0': 0.0097, 'pts_bbox_NuScenes/construction_vehicle_trans_err': 1.0013, 'pts_bbox_NuScenes/construction_vehicle_scale_err': 0.5295, 'pts_bbox_NuScenes/construction_vehicle_orient_err': 1.144, 'pts_bbox_NuScenes/construction_vehicle_vel_err': 0.1305, 'pts_bbox_NuScenes/construction_vehicle_attr_err': 0.2276, 'pts_bbox_NuScenes/bus_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/bus_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/bus_AP_dist_2.0': 0.0076, 'pts_bbox_NuScenes/bus_AP_dist_4.0': 0.0606, 'pts_bbox_NuScenes/bus_trans_err': 1.1117, 'pts_bbox_NuScenes/bus_scale_err': 0.2478, 'pts_bbox_NuScenes/bus_orient_err': 0.5546, 'pts_bbox_NuScenes/bus_vel_err': 2.4679, 'pts_bbox_NuScenes/bus_attr_err': 0.6041, 'pts_bbox_NuScenes/trailer_AP_dist_0.5': 0.0, 'pts_bbox_NuScenes/trailer_AP_dist_1.0': 0.0, 'pts_bbox_NuScenes/trailer_AP_dist_2.0': 0.0116, 'pts_bbox_NuScenes/trailer_AP_dist_4.0': 0.072, 'pts_bbox_NuScenes/trailer_trans_err': 1.1001, 'pts_bbox_NuScenes/trailer_scale_err': 0.283, 'pts_bbox_NuScenes/trailer_orient_err': 0.908, 'pts_bbox_NuScenes/trailer_vel_err': 0.3297, 'pts_bbox_NuScenes/trailer_attr_err': 0.1348, 'pts_bbox_NuScenes/barrier_AP_dist_0.5': 0.011, 'pts_bbox_NuScenes/barrier_AP_dist_1.0': 0.0915, 'pts_bbox_NuScenes/barrier_AP_dist_2.0': 0.2187, 'pts_bbox_NuScenes/barrier_AP_dist_4.0': 0.3434, 'pts_bbox_NuScenes/barrier_trans_err': 0.8034, 'pts_bbox_NuScenes/barrier_scale_err': 0.3798, 'pts_bbox_NuScenes/barrier_orient_err': 0.3595, 'pts_bbox_NuScenes/barrier_vel_err': nan, 'pts_bbox_NuScenes/barrier_attr_err': nan, 'pts_bbox_NuScenes/motorcycle_AP_dist_0.5': 0.0253, 'pts_bbox_NuScenes/motorcycle_AP_dist_1.0': 0.133, 'pts_bbox_NuScenes/motorcycle_AP_dist_2.0': 0.2365, 'pts_bbox_NuScenes/motorcycle_AP_dist_4.0': 0.3129, 'pts_bbox_NuScenes/motorcycle_trans_err': 0.6813, 'pts_bbox_NuScenes/motorcycle_scale_err': 0.2961, 'pts_bbox_NuScenes/motorcycle_orient_err': 0.629, 'pts_bbox_NuScenes/motorcycle_vel_err': 1.0745, 'pts_bbox_NuScenes/motorcycle_attr_err': 0.1937, 'pts_bbox_NuScenes/bicycle_AP_dist_0.5': 0.0458, 'pts_bbox_NuScenes/bicycle_AP_dist_1.0': 0.1917, 'pts_bbox_NuScenes/bicycle_AP_dist_2.0': 0.3496, 'pts_bbox_NuScenes/bicycle_AP_dist_4.0': 0.4252, 'pts_bbox_NuScenes/bicycle_trans_err': 0.6635, 'pts_bbox_NuScenes/bicycle_scale_err': 0.311, 'pts_bbox_NuScenes/bicycle_orient_err': 0.7029, 'pts_bbox_NuScenes/bicycle_vel_err': 0.4081, 'pts_bbox_NuScenes/bicycle_attr_err': 0.0378, 'pts_bbox_NuScenes/pedestrian_AP_dist_0.5': 0.1057, 'pts_bbox_NuScenes/pedestrian_AP_dist_1.0': 0.264, 'pts_bbox_NuScenes/pedestrian_AP_dist_2.0': 0.3596, 'pts_bbox_NuScenes/pedestrian_AP_dist_4.0': 0.4512, 'pts_bbox_NuScenes/pedestrian_trans_err': 0.5412, 'pts_bbox_NuScenes/pedestrian_scale_err': 0.3233, 'pts_bbox_NuScenes/pedestrian_orient_err': 0.6827, 'pts_bbox_NuScenes/pedestrian_vel_err': 0.649, 'pts_bbox_NuScenes/pedestrian_attr_err': 0.3156, 'pts_bbox_NuScenes/traffic_cone_AP_dist_0.5': 0.0235, 'pts_bbox_NuScenes/traffic_cone_AP_dist_1.0': 0.1444, 'pts_bbox_NuScenes/traffic_cone_AP_dist_2.0': 0.2996, 'pts_bbox_NuScenes/traffic_cone_AP_dist_4.0': 0.4316, 'pts_bbox_NuScenes/traffic_cone_trans_err': 0.7734, 'pts_bbox_NuScenes/traffic_cone_scale_err': 0.3937, 'pts_bbox_NuScenes/traffic_cone_orient_err': nan, 'pts_bbox_NuScenes/traffic_cone_vel_err': nan, 'pts_bbox_NuScenes/traffic_cone_attr_err': nan, 'pts_bbox_NuScenes/NDS': 0.27776341220635314, 'pts_bbox_NuScenes/mAP': 0.13476739579111732}

Thanks in advance :)

yanwei-li commented 1 year ago

Hi, You can try v0.17.3 for the inference. I guess the mismatch is caused by the coordinate change in the new version. Of course, if you train it using v1.0.0rc5, it should achieve the correct results. Thanks!