search

Sranc3 / M-BEV

[AAAI24]This is the implementation for the paper M-BEV: Masked BEV Perception for Robust Autonomous Driving
32 stars 4 forks source link

Performance of the release model #6

Closed Daniel-xsy closed 5 months ago

Daniel-xsy commented 5 months ago

Thanks for your great work. Could you provide the results of the released mbev.pth checkpoints? I tested it and found it's lower than that reported in the paper. Below is my results:

mAP: 0.3525
mATE: 0.7259
mASE: 0.7081
mAOE: 1.5453
mAVE: 0.4288
mAAE: 0.1856
NDS: 0.3714
Eval time: 111.4s

Per-class results:
Object Class    AP      ATE     ASE     AOE     AVE     AAE
car     0.530   0.546   0.748   1.581   0.398   0.181
truck   0.315   0.754   0.794   1.600   0.389   0.207
bus     0.370   0.742   0.856   1.549   0.771   0.257
trailer 0.194   1.077   0.841   1.574   0.294   0.137
construction_vehicle    0.102   0.960   0.694   1.527   0.145   0.343
pedestrian      0.407   0.694   0.335   1.530   0.488   0.197
motorcycle      0.320   0.699   0.794   1.534   0.647   0.154
bicycle 0.318   0.593   0.806   1.722   0.298   0.008
traffic_cone    0.504   0.556   0.327   nan     nan     nan
barrier 0.466   0.638   0.887   1.289   nan     nan
Sranc3 commented 5 months ago

It's probably the problem with the mmdet3d version, the mASE and mAOE mertics are misaligned and I have met this before. This should be solved using the right version of mmdet3d.

Daniel-xsy commented 5 months ago

Thanks for your reply, could you provide the mmdet3d version you use? I currently use 0.17.1 as BEVFormer casebase.

Sranc3 commented 5 months ago

I think this could be helpful 'https://github.com/megvii-research/PETR/issues/128', and for our experiment we use the new version mmdet3d 1.0.

Daniel-xsy commented 5 months ago

Thanks!