[Bug] Lyft Dataset Evaluation Precision Drop

ZLTJohn commented 1 year ago

Prerequisite

[X] I have searched Issues and Discussions but cannot get the expected help.
[X] I have read the FAQ documentation but cannot get the expected help.
[X] The bug has not been fixed in the latest version (dev) or latest version (1.x).

Task

I'm using the official example scripts/configs for the officially supported tasks/models/datasets.

Branch

1.1x branch https://github.com/open-mmlab/mmdetection3d/tree/1.1

Environment

sys.platform: linux Python: 3.8.15 (default, Nov 24 2022, 15:19:38) [GCC 11.2.0] CUDA available: True numpy_random_seed: 2147483648 GPU 0,1,2,3,4,5,6,7: GeForce RTX 3080 CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 11.2, V11.2.142 GCC: gcc (GCC) 5.4.0 PyTorch: 1.9.0+cu111 PyTorch compiling details: PyTorch built with:

GCC 7.3
C++ Version: 201402
Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
Intel(R) MKL-DNN v2.1.2 (Git Hash 98be7e8afa711dc9b66c8ff3504129cb82013cdb)
OpenMP 201511 (a.k.a. OpenMP 4.5)
NNPACK is enabled
CPU capability usage: AVX2
CUDA Runtime 11.1
NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
CuDNN 8.0.5
Magma 2.5.2
Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.9.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

TorchVision: 0.10.0+cu111 OpenCV: 4.6.0 MMEngine: 0.6.0 MMDetection: 3.0.0rc4 MMDetection3D: 1.1.0rc3+713d7e6 spconv2.0: False

Reproduces the problem - code sample

I generate lyft_l5 dataset based on the latest mmdet3d dataset converter. I followed readme.md to test my newly generated lyft_l5 dataset, and I used the second line: FPN | 2x | 9.2 | | 14.8 | 15.0 | model | log

Reproduces the problem - command or script

bash tools/dist_test.sh configs/pointpillars/pointpillars_hv_fpn_sbn-all_8xb2-2x_lyft-3d.py checkpoints/hv_pointpillars_fpn_sbn-all_2x8_2x_lyft-3d_20210822_095429-0b3d6196.pth 8

Reproduces the problem - error message

My result: Lyft metric/pred_instances_3d_Lyft/animal_AP: 0.0000
Lyft metric/pred_instances_3d_Lyft/bicycle_AP: 0.0685
Lyft metric/pred_instances_3d_Lyft/bus_AP: 0.1532
Lyft metric/pred_instances_3d_Lyft/car_AP: 0.3830
Lyft metric/pred_instances_3d_Lyft/emergency_vehicle_AP: 0.0000 Lyft metric/pred_instances_3d_Lyft/motorcycle_AP: 0.0177
Lyft metric/pred_instances_3d_Lyft/other_vehicle_AP: 0.3158
Lyft metric/pred_instances_3d_Lyft/pedestrian_AP: 0.0490
Lyft metric/pred_instances_3d_Lyft/truck_AP: 0.1165
Lyft metric/pred_instances_3d_Lyft/mAP: 0.1226

Results in the corresponding log: "pts_bbox_Lyft/animal_AP": 0.0, "pts_bbox_Lyft/bicycle_AP": 0.07395, "pts_bbox_Lyft/bus_AP": 0.15684, "pts_bbox_Lyft/car_AP": 0.39534, "pts_bbox_Lyft/emergency_vehicle_AP": 0.0, "pts_bbox_Lyft/motorcycle_AP": 0.01702, "pts_bbox_Lyft/other_vehicle_AP": 0.32052, "pts_bbox_Lyft/pedestrian_AP": 0.05486, "pts_bbox_Lyft/truck_AP": 0.12023, "pts_bbox_Lyft/mAP": 0.12653

Additional information

Note that 'bbox3d = detection['bbox_3d']' will cause error in lyft_metric.py and the correct one should be bbox3d = detection['bboxes_3d']

Xiangxu-0103 commented 1 year ago

Thanks for your reminder. It seems that the checkpoint you used is trained with mmdet3d v0.x, which means before our coordinate refactor, and the latest checkpoint has not been updated.

ZLTJohn commented 1 year ago

OK, and maybe we can fix the bug in lyft metric

Xiangxu-0103 commented 1 year ago

OK, and maybe we can fix the bug in lyft metric

Are you willing to create a PR to help us fix the bug?

open-mmlab / mmdetection3d