[Bug] FCOS3D validation with input shape [512, 768], mAP=0

Hukongtao commented 1 year ago

Prerequisite

[X] I have searched Issues and Discussions but cannot get the expected help.
[X] I have read the FAQ documentation but cannot get the expected help.
[X] The bug has not been fixed in the latest version (dev) or latest version (1.x).

Task

I'm using the official example scripts/configs for the officially supported tasks/models/datasets.

Branch

master branch https://github.com/open-mmlab/mmdetection3d

Environment

sys.platform: linux Python: 3.8.12 (default, Aug 9 2022, 19:33:50) [GCC 5.4.0] CUDA available: True GPU 0,1,2,3: NVIDIA TITAN V CUDA_HOME: /usr/local/cuda-11.1 NVCC: Cuda compilation tools, release 11.1, V11.1.74 GCC: gcc (GCC) 7.3.1 20180303 (Red Hat 7.3.1-5) PyTorch: 1.10.2+cu111 PyTorch compiling details: PyTorch built with:

GCC 7.3
C++ Version: 201402
Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)
OpenMP 201511 (a.k.a. OpenMP 4.5)
LAPACK is enabled (usually provided by MKL)
NNPACK is enabled
CPU capability usage: AVX2
CUDA Runtime 11.1
NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
CuDNN 8.0.5
Magma 2.5.2
Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.10.2, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

TorchVision: 0.11.3+cu111 OpenCV: 4.6.0 MMCV: 1.7.0 MMCV Compiler: GCC 5.4 MMCV CUDA Compiler: 11.1 MMDetection: 2.25.3 MMSegmentation: 0.29.1 MMDetection3D: 1.0.0rc5+962fc83 spconv2.0: True

Reproduces the problem - code sample

First, you should change the test_pipeline in the config to,

test_pipeline = [
    dict(type='LoadImageFromFileMono3D'),
    dict(type='Resize', img_scale=(768, 512), keep_ratio=True),
    dict(type='Normalize', **img_norm_cfg),
    dict(type='Pad', size_divisor=32),
    dict(type='DefaultFormatBundle3D', class_names=class_names, with_label=False),
    dict(type='Collect3D', keys=['img']),
]

then, you should change the code https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/models/dense_heads/fcos_mono3d_head.py#L641 to:

if rescale:
    bbox_pred[:, :2] /= bbox_pred[:, :2].new_tensor(scale_factor[0])
    bbox_pred[:, 3:6] /= bbox_pred[:, 3:6].new_tensor(scale_factor[0])

Then you can run the validation:

python tools/test.py config configs/fcos3d/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d.py checkpoint checkpoints/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d_finetune_20210717_095645-8d806dc2.pth --out outputs/resize_result.pickle --gpu-id 0 --eval mAP

Finally, you get:

mAP: 0.0026
mATE: 1.0238
mASE: 0.9864
mAOE: 0.9358
mAVE: 1.0000
mAAE: 1.0000
NDS: 0.0091
Eval time: 72.5s

Reproduces the problem - command or script

With less resolution, the mAP should be lower. but should not be 0.

Reproduces the problem - error message

No error, Just the mAP is low,and I don't know why.

Additional information

No response

Hukongtao commented 1 year ago

@Tai-Wang Can you help me with this question?

Tai-Wang commented 1 year ago

It's because monocular 3D detectors are sensitive to the change of input images. When the input resolution is changed, at least we need to allow the network to see them during the training phase. Actually, our pre-release version can support this augmentation and we will release this feature together with some others new in about 1-2 months.

Hukongtao commented 1 year ago

It's because monocular 3D detectors are sensitive to the change of input images. When the input resolution is changed, at least we need to allow the network to see them during the training phase. Actually, our pre-release version can support this augmentation and we will release this feature together with some others new in about 1-2 months.

Thank you very much I will try this resize3D right away.

Hukongtao commented 1 year ago

@Tai-Wang https://github.com/open-mmlab/mmdetection3d/blob/master/configs/fcos3d/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d.py#L27 In fact, I think the resize here will cause some misunderstandings for users, making users think that this resize can be used for training

Tai-Wang commented 1 year ago

@Tai-Wang https://github.com/open-mmlab/mmdetection3d/blob/master/configs/fcos3d/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d.py#L27 In fact, I think the resize here will cause some misunderstandings for users, making users think that this resize can be used for training

Yes, thanks for your suggestions. We may consider adding some comments to avoid such confusion.

open-mmlab / mmdetection3d