search

open-mmlab / mmdetection3d

OpenMMLab's next-generation platform for general 3D object detection.
https://mmdetection3d.readthedocs.io/en/latest/
Apache License 2.0
5.16k stars 1.52k forks source link

PETR metric issues under version mmdetection3d-1.1.0 #2510

Open xiaowuge1201 opened 1 year ago

xiaowuge1201 commented 1 year ago

I trained PETR 24epoch through A800, but I cannot achieve the accuracy you announced. The metric I received is： mAP: 0.3131 mATE: 0.9197 mASE: 0.2800 mAOE: 0.6155 mAVE:0.8982 mAAE:0.2354 NDS: 0.3616 Can you publish the training log？？？

SekiroRong commented 1 year ago

Here's my training log: 20221224_160645.log

xiaowuge1201 commented 1 year ago

There is a problem with the Petr in the current mmdet3d-1.1.0 version, which may cause errors during runtime. Have you encountered this situation? You can use this version of code to run and test it; I made some minor corrections to the errors that occurred during the operation and conducted training, but the results were much worse than the ones you trained last year. Here is my training log： 20230509_123941.log， Can you use this version for retraining at your convenience and submit the log for comparison? Thank you

Here's my training log: 20221224_160645.log

SekiroRong commented 1 year ago

Could you please list the changes and errors? Unfortunately, I don't have the computation source to retrain PETR right now. However, you might be able to use the same version of this PETR implementation that I committed.

JingweiZhang12 commented 1 year ago

@xiaowuge1201 The links of the log and model have been uploaded in README.md in dev-1.x branch.

carlinds commented 6 months ago

Hi! I also experience issues with reproducing the reported metrics. For reference, I get the following metrics when running a test on nuScenes validation split (with the config and model in the README):

mAP: 0.3829
mATE: 0.7376
mASE: 0.7096
mAOE: 0.4803
mAVE: 0.8703
mAAE: 0.2040
NDS: 0.3913

As you can see, the metrics differ from the reported ones, especially for mASE (and in turn NDS). The log you refer to seems to be from a much older state of the repo (it contains the now removed LidarBox3dVersionTransfrom for instance). It seems plausible that something has gone wrong when adapting the code for the coordinate system refactoring.

Did you run any model training and/or testing related to the following commits? https://github.com/open-mmlab/mmdetection3d/commit/952a59235771c9226b8f5016d3d8e615b8eb298b https://github.com/open-mmlab/mmdetection3d/commit/12c3b19d8c788111bf66a375400e768c2790b3a1