search

tudelft-iv / UniBEV

[IV'24] UniBEV: the official implementation of UniBEV
16 stars 3 forks source link

Reproduce Issue #3

Closed Daniel-xsy closed 2 weeks ago

Daniel-xsy commented 3 weeks ago

Dear author, thanks for your interesting work.

I tried to reproduce the results reported in the paper using the configuration file unibev_cnw_dim_256_nus_LC_full. However, I ended up with a noticeably lower NDS compared to the paper. Especially, I get a very high mAOEand mASR error. The results are as follows:

Evaluating bboxes of pts_bbox
mAP: 0.6246                                                                                      
mATE: 0.3273
mASE: 0.7101
mAOE: 1.5555
mAVE: 0.3135
mAAE: 0.1814
NDS: 0.5591

Furthermore, I tried to fine-tune the provided checkpoint file unibev_cnw_256_nus_MD.pth, and the same problem popped up. Could you share any information on why this is happening?

Shiming94 commented 3 weeks ago

Hi, thank you so much for your interest in our work. The main issue is we use different versions of the mmdet3d data preprocess script.

Please refer to this link for more details. The main modification is the yaw angle definition of bounding boxes, which will cause a significant increase in the orientation error.

So that you know, we use mmdet3d v0.18.1; any version newer than v0.18.1 will cause this problem. We are also working on an updated version based on the latest mmdetection3d, but we are still not sure when it will be finished.

Shiming94 commented 3 weeks ago

Hey, I will close this issue. Should you have further questions, you can either reopen it or open a new issue.

Daniel-xsy commented 3 weeks ago

Hi @Shiming94, Thanks for your reply! Actually, I was using mmdet3d v0.18.1:

Python: 3.7.16 (default, Jan 17 2023, 22:20:44) [GCC 11.2.0]
CUDA available: True
GPU 0,1,2,3,4,5,6,7: NVIDIA A800-SXM4-80GB
CUDA_HOME: /usr/local/cuda
NVCC: Build cuda_11.8.r11.8/compiler.31833905_0
GCC: gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
PyTorch: 1.10.0+cu111
PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX512
  - CUDA Runtime 11.1
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
  - CuDNN 8.0.5
  - Magma 2.5.2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.10.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, 

TorchVision: 0.11.0+cu111
OpenCV: 4.10.0
MMCV: 1.3.17
MMCV Compiler: GCC 7.3
MMCV CUDA Compiler: 11.1
MMDetection: 2.19.0
MMSegmentation: 0.20.0
MMDetection3D: 0.18.1+f459770

Do you use any specific modification in the codebase to pre-process the yaw angle in the training phase? I briefly checked the codebase and found no of these kinds of pre-processing customization.

I also use the same nuscenes_infos_train.pkl and nuscenes_infos_val.pkl for other codebase, so I guess it might not be the issue of .pkl files.

Thanks for your attention.

Shiming94 commented 2 weeks ago

Hi, Shaoyuan @Daniel-xsy, it is weird... I didn't have any other data processing in the training phase. Are your nuscenes_infos_train.pkl and nuscenes_infos_val.pkl also generated with create_data.py under version 0.18.1? Late versions of mmdet3d also changed the definition of yaw angle in the annotation file.

Daniel-xsy commented 2 weeks ago

Yes...since I can reproduce the provided checkpoint file, I think the nuscenes_infos_val.pkl should be fine, I will double check the nuscenes_infos_train.pkl file. Thanks for your patient reply anyway!

Shiming94 commented 2 weeks ago

Hi, thank you so much! So you can reproduce the number in the table with the validation data, right? Then I think it should be the version mismatch of training data file. Please keep me updated!

I will close this issue, you can reopen it anytime if you have further questions.