Open weinixiong8964 opened 6 months ago
The cam-type is somehow failed to pass its value here. I add cam_type = 'CAM0'
before img_path = data_info['images'][cam_type]['img_path']
. It passed, while another problem raised:
KeyError: 'cam2img'
So I output the content of demo/data/sunrgbd/sunrgbd_000017_infos.pkl
and found that there is no cam2img. It seems like intrinsics for a camera or something.
I ran this on kitti too. It lack of a parameter called depth2img, whereas pkl file of sunrgbd has.
Wired.
I solved it. Just as I posted before, some important parameters were missing. Some changes:
in mmdetection3d/mmdet3d/apis/inferencers/multi_modality_det3d_inferencer.py
(Ln 117, 125, and etc.), it failed to aquire inner and extrinsics, I directly get them in SUNRGBD dataset(intrinsics.txt and extrinsics/20151113224133.txt):
# cam2img = np.asarray(
# data_info['images'][cam_type]['cam2img'], dtype=np.float32)
# lidar2cam = np.asarray(
# data_info['images'][cam_type]['lidar2cam'],
# dtype=np.float32)
# if 'lidar2img' in data_info['images'][cam_type]:
# lidar2img = np.asarray(
# data_info['images'][cam_type]['lidar2img'],
# dtype=np.float32)
# else:
# lidar2img = cam2img @ lidar2cam
# lidar2img = cam2img @ lidar2cam[0:3]
# for sunrgbd
cam2img = np.asarray([[574.334300, 0.000000, 320.000000, 0.000000],
[0.000000, 574.334300, 240.000000, 0.000000],
[0.000000, 0.000000, 1.000000, 0.000000],
[0.000000, 0.000000, 0.000000, 1.000000]]
)
lidar2cam = np.asarray(
[[0.999997, 0.002445, 0.000612, 0.000000],
[-0.002445, 0.881972, 0.471295, 0.000000],
[0.000612, -0.471295, 0.881975, 0.000000],
[0, 0, 0, 1]]
)
lidar2img = cam2img @ lidar2cam
input['cam2img'] = cam2img
input['lidar2cam'] = lidar2cam
input['lidar2img'] = lidar2img
in mmdetection3d/mmdet3d/models/layers/fusion_layers/vote_fusion.py
, it failed to aquire depth to img parameter. I directly wrote the data in sunrgbd (ln 62) here:
depth2img = torch.tensor([[530.5366320782341, 362.0406589161139, -32.432176765054464], [35.10014939727262, 164.8833093419671, -567.605659916997], [0.005785000044852495, 0.9835000038146973, -0.18082000315189362]]).cuda()
# depth2img = xyz_depth.new_tensor(img_meta['depth2img'])
Prerequisite
Task
I'm using the official example scripts/configs for the officially supported tasks/models/datasets.
Branch
main branch https://github.com/open-mmlab/mmdetection3d
Environment
sys.platform: linux Python: 3.8.19 (default, Mar 20 2024, 19:58:24) [GCC 11.2.0] CUDA available: True MUSA available: False numpy_random_seed: 2147483648 GPU 0,1: NVIDIA GeForce RTX 4090 D CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 11.3, V11.3.109 GCC: gcc (Ubuntu 9.5.0-1ubuntu1~22.04) 9.5.0 PyTorch: 1.12.0+cu113 PyTorch compiling details: PyTorch built with:
TorchVision: 0.13.0+cu113 OpenCV: 4.9.0 MMEngine: 0.10.4 MMDetection: 3.1.0 MMDetection3D: 1.4.0+ spconv2.0: True
Reproduces the problem - code sample
python demo/mono_det_demo.py demo/data/nuscenes/n015-2018-07-24-11-22-45+0800__CAM_BACK__1532402927637525.jpg demo/data/nuscenes/n015-2018-07-24-11-22-45+0800.pkl configs/fcos3d/fcos3d_r101-caffe-dcn_fpn_head-gn_8xb2-1x_nus-mono3d_finetune.py ${CHECKPOINT_FILE} --show --cam-type CAM_BACK
Reproduces the problem - command or script
python demo/mono_det_demo.py demo/data/nuscenes/n015-2018-07-24-11-22-45+0800__CAM_BACK__1532402927637525.jpg demo/data/nuscenes/n015-2018-07-24-11-22-45+0800.pkl configs/fcos3d/fcos3d_r101-caffe-dcn_fpn_head-gn_8xb2-1x_nus-mono3d_finetune.py fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d_finetune_20210717_095645-8d806dc2.pth --show --cam-type CAM_BACK
Reproduces the problem - error message
This may cause unexpected failure when running the built modules. Please check whether "mmdet3d" is a correct scope, or whether the registry is initialized. /home/yuanyue2021/anaconda3/envs/mm3d/lib/python3.8/site-packages/mmengine/visualization/visualizer.py:196: UserWarning: Failed to add <class 'mmengine.visualization.vis_backend.LocalVisBackend'>, please provide the
main()
File "demo/mono_det_demo.py", line 88, in main
inferencer(**call_args)
File "/mnt/DataDiskA/yuanyue2021/MMDetection/mmdetection3d-new/mmdet3d/apis/inferencers/base_3d_inferencer.py", line 204, in call
ori_inputs = self._inputs_to_list(inputs, cam_type=cam_type)
File "/mnt/DataDiskA/yuanyue2021/MMDetection/mmdetection3d-new/mmdet3d/apis/inferencers/mono_det3d_inferencer.py", line 110, in _inputs_to_list
img_path = data_info['images'][cam_type]['img_path']
KeyError: 'CAM2'
save_dir
argument. warnings.warn(f'Failed to add {vis_backend.class}, ' Traceback (most recent call last): File "demo/mono_det_demo.py", line 98, inAdditional information
No response