open-mmlab / mmdetection3d

OpenMMLab's next-generation platform for general 3D object detection.
https://mmdetection3d.readthedocs.io/en/latest/
Apache License 2.0
5.31k stars 1.54k forks source link

[Bug] img_path = data_info['images'][cam_type]['img_path'] KeyError: 'CAM2' #2981

Open weinixiong8964 opened 6 months ago

weinixiong8964 commented 6 months ago

Prerequisite

Task

I'm using the official example scripts/configs for the officially supported tasks/models/datasets.

Branch

main branch https://github.com/open-mmlab/mmdetection3d

Environment

sys.platform: linux Python: 3.8.19 (default, Mar 20 2024, 19:58:24) [GCC 11.2.0] CUDA available: True MUSA available: False numpy_random_seed: 2147483648 GPU 0,1: NVIDIA GeForce RTX 4090 D CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 11.3, V11.3.109 GCC: gcc (Ubuntu 9.5.0-1ubuntu1~22.04) 9.5.0 PyTorch: 1.12.0+cu113 PyTorch compiling details: PyTorch built with:

TorchVision: 0.13.0+cu113 OpenCV: 4.9.0 MMEngine: 0.10.4 MMDetection: 3.1.0 MMDetection3D: 1.4.0+ spconv2.0: True

Reproduces the problem - code sample

python demo/mono_det_demo.py demo/data/nuscenes/n015-2018-07-24-11-22-45+0800__CAM_BACK__1532402927637525.jpg demo/data/nuscenes/n015-2018-07-24-11-22-45+0800.pkl configs/fcos3d/fcos3d_r101-caffe-dcn_fpn_head-gn_8xb2-1x_nus-mono3d_finetune.py ${CHECKPOINT_FILE} --show --cam-type CAM_BACK

Reproduces the problem - command or script

python demo/mono_det_demo.py demo/data/nuscenes/n015-2018-07-24-11-22-45+0800__CAM_BACK__1532402927637525.jpg demo/data/nuscenes/n015-2018-07-24-11-22-45+0800.pkl configs/fcos3d/fcos3d_r101-caffe-dcn_fpn_head-gn_8xb2-1x_nus-mono3d_finetune.py fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d_finetune_20210717_095645-8d806dc2.pth --show --cam-type CAM_BACK

Reproduces the problem - error message

This may cause unexpected failure when running the built modules. Please check whether "mmdet3d" is a correct scope, or whether the registry is initialized. /home/yuanyue2021/anaconda3/envs/mm3d/lib/python3.8/site-packages/mmengine/visualization/visualizer.py:196: UserWarning: Failed to add <class 'mmengine.visualization.vis_backend.LocalVisBackend'>, please provide the save_dir argument. warnings.warn(f'Failed to add {vis_backend.class}, ' Traceback (most recent call last): File "demo/mono_det_demo.py", line 98, in main() File "demo/mono_det_demo.py", line 88, in main inferencer(**call_args) File "/mnt/DataDiskA/yuanyue2021/MMDetection/mmdetection3d-new/mmdet3d/apis/inferencers/base_3d_inferencer.py", line 204, in call ori_inputs = self._inputs_to_list(inputs, cam_type=cam_type) File "/mnt/DataDiskA/yuanyue2021/MMDetection/mmdetection3d-new/mmdet3d/apis/inferencers/mono_det3d_inferencer.py", line 110, in _inputs_to_list img_path = data_info['images'][cam_type]['img_path'] KeyError: 'CAM2'

Additional information

No response

KopiSoftware commented 4 months ago

The cam-type is somehow failed to pass its value here. I add cam_type = 'CAM0' before img_path = data_info['images'][cam_type]['img_path']. It passed, while another problem raised: KeyError: 'cam2img' So I output the content of demo/data/sunrgbd/sunrgbd_000017_infos.pkl and found that there is no cam2img. It seems like intrinsics for a camera or something. I ran this on kitti too. It lack of a parameter called depth2img, whereas pkl file of sunrgbd has. Wired.

KopiSoftware commented 4 months ago

I solved it. Just as I posted before, some important parameters were missing. Some changes: in mmdetection3d/mmdet3d/apis/inferencers/multi_modality_det3d_inferencer.py (Ln 117, 125, and etc.), it failed to aquire inner and extrinsics, I directly get them in SUNRGBD dataset(intrinsics.txt and extrinsics/20151113224133.txt):

                # cam2img = np.asarray(
                #     data_info['images'][cam_type]['cam2img'], dtype=np.float32)
                # lidar2cam = np.asarray(
                #     data_info['images'][cam_type]['lidar2cam'],
                #     dtype=np.float32)
                # if 'lidar2img' in data_info['images'][cam_type]:
                #     lidar2img = np.asarray(
                #         data_info['images'][cam_type]['lidar2img'],
                #         dtype=np.float32)
                # else:
                #     lidar2img = cam2img @ lidar2cam
                    # lidar2img = cam2img @ lidar2cam[0:3]

                # for sunrgbd
                cam2img = np.asarray([[574.334300, 0.000000, 320.000000, 0.000000],
                                      [0.000000, 574.334300, 240.000000, 0.000000],
                                      [0.000000, 0.000000, 1.000000, 0.000000],
                                      [0.000000, 0.000000, 0.000000, 1.000000]]
                                    )
                lidar2cam = np.asarray(
                    [[0.999997, 0.002445, 0.000612, 0.000000],
                    [-0.002445, 0.881972, 0.471295, 0.000000],
                    [0.000612, -0.471295, 0.881975, 0.000000],
                    [0, 0, 0, 1]]
                )

                lidar2img = cam2img @ lidar2cam

                input['cam2img'] = cam2img
                input['lidar2cam'] = lidar2cam
                input['lidar2img'] = lidar2img

in mmdetection3d/mmdet3d/models/layers/fusion_layers/vote_fusion.py, it failed to aquire depth to img parameter. I directly wrote the data in sunrgbd (ln 62) here:

            depth2img = torch.tensor([[530.5366320782341, 362.0406589161139, -32.432176765054464], [35.10014939727262, 164.8833093419671, -567.605659916997], [0.005785000044852495, 0.9835000038146973, -0.18082000315189362]]).cuda()
            # depth2img = xyz_depth.new_tensor(img_meta['depth2img'])