nuscenes 3D框投影到图片上可视化的问题

你好 @junjie18 ！我尝试将3D框投影到图像上，也就是类似你论文中的Figure 5.那样。

尝试过程：我通过官方的tools/misc/visualize_results.py来实现的，主要重写了CustomNuScenesDataset类中的show()方法，

` class CustomNuScenesDataset(NuScenesDataset): .......

def show(self, results, out_dir, show=False, pipeline=None): """Results visualization.

    Args:
        results (list[dict]): List of bounding boxes results.
        out_dir (str): Output directory of visualization result.
        show (bool): Whether to visualize the results online.
            Default: False.
        pipeline (list[dict], optional): raw data loading for showing.
            Default: None.
    """
    assert out_dir is not None, 'Expect out_dir, got none.'
    pipeline = self._get_pipeline(pipeline)
    for i, result in enumerate(results):
        # 绘制点云和3D框
        if 'pts_bbox' in result.keys():
            result = result['pts_bbox']
        data_info = self.data_infos[i]
        pts_path = data_info['lidar_path']
        file_name = osp.split(pts_path)[-1].split('.')[0]
        points = self._extract_data(i, pipeline, 'points').numpy()
        # for now we convert points into depth mode
        points = Coord3DMode.convert_point(points, Coord3DMode.LIDAR,
                                           Coord3DMode.DEPTH)
        inds = result['scores_3d'] > 0.1
        gt_bboxes = self.get_ann_info(i)['gt_bboxes_3d'].tensor.numpy()
        show_gt_bboxes = Box3DMode.convert(gt_bboxes, Box3DMode.LIDAR,
                                           Box3DMode.DEPTH)
        pred_bboxes = result['boxes_3d'][inds].tensor.numpy()
        show_pred_bboxes = Box3DMode.convert(pred_bboxes, Box3DMode.LIDAR,
                                             Box3DMode.DEPTH)
        show_result(points, show_gt_bboxes, show_pred_bboxes, out_dir,
                    file_name, show)

        # 多模态可视化, 在图片上可视化投影 3D 框 
        if 'img_bbox' in result.keys():
            result = result['img_bbox']
        data_info = self.data_infos[i]

        img, img_metas = self._extract_data(i, pipeline,
                                            ['img', 'img_metas'])
        img_path = img_metas['filename']
        file_name = [ osp.split(tmp)[-1].split('.')[0] for tmp in img_path]

        # need to transpose channel to first dim
        img = img.numpy().transpose(0, 2, 3, 1)         # [view, c, w, h] -> [view, w, h, c]

        gt_bboxes = self.get_ann_info(i)['gt_bboxes_3d']
        pred_bboxes = result['boxes_3d']

        for j in range(img.shape[0]):               # for 6-views
            show_multi_modality_result(
                img[j, ...],
                gt_bboxes,
                pred_bboxes,
                img_metas['lidar2img'][j],
                out_dir,
                file_name[j],
                box_mode='lidar',
                show=False)              # false因为没有GUI界面来展示

对于show()方法的后半段，“在图片上可视化投影 3D 框” 中，show_multi_modality_result()最后在绘制pred_bboxes3D投影图时，cv2画图时报错：

Exception has occurred: error (note: full exception trace is shown but execution is paused at: _run_module_as_main) OpenCV(4.8.0) :-1: error: (-5:Bad argument) in function 'line'

Overload resolution failed:

Can't parse 'pt1'. Sequence item with index 0 has a wrong type

Can't parse 'pt1'. Sequence item with index 0 has a wrong type File "/opt/conda/lib/python3.7/site-packages/mmdet3d/core/visualizer/image_vis.py", line 88, in plot_rect3d_on_img cv2.LINE_AA) File "/opt/conda/lib/python3.7/site-packages/mmdet3d/core/visualizer/image_vis.py", line 128, in draw_lidar_bbox3d_on_img return plot_rect3d_on_img(img, num_bbox, imgfov_pts_2d, color, thickness) File "/opt/conda/lib/python3.7/site-packages/mmdet3d/core/visualizer/show_result.py", line 290, in show_multi_modality_result pred_bboxes, img, proj_mat, img_metas, color=pred_bbox_color) File "/3dv/CMT/projects/mmdet3d_plugin/datasets/custom_nuscenes_dataset.py", line 176, in show show=False)
File "/3dv/CMT/tools/result_models.py", line 63, in main dataset.show(results, args.show_dir, pipeline=eval_pipeline) File "/3dv/CMT/tools/result_models.py", line 74, in main() File "/opt/conda/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/opt/conda/lib/python3.7/runpy.py", line 193, in _run_module_as_main (Current frame) "main", mod_spec) cv2.error: OpenCV(4.8.0) :-1: error: (-5:Bad argument) in function 'line' Overload resolution failed:

Can't parse 'pt1'. Sequence item with index 0 has a wrong type

Can't parse 'pt1'. Sequence item with index 0 has a wrong type

我发现，传入的gt_bboxes.corners和pred_bboxes.corners包含了大量的负数，不知道是不是这方面的问题，但是gt_bboxes能画出来(只是好像是在特征图上画的)，pred_bboxes就会报错？

期待回复，请大佬多多指教！！

@James-S-choi - I am facing the same issue, could you please let whether you were able to fix this issue ? I can able to visualize the point cloud outputs, but the show_multi_modality_result() is giving me the same error. Kindly let me know how did you manage to fix it ? Thanks @junjie18 kindly share your thoughts

  File "tools/test.py", line 313, in <module>
    main()
  File "tools/test.py", line 287, in main
    dataset.show(outputs, args.show_dir, pipeline=eval_pipeline)
  File "/media/ava/Data_CI/selventh/eX-AI/demos/CMT/projects/mmdet3d_plugin/datasets/custom_nuscenes_dataset.py", line 143, in show
    show_multi_modality_result(
  File "/home/ava/anaconda3/envs/selventh/lib/python3.8/site-packages/mmdet3d/core/visualizer/show_result.py", line 289, in show_multi_modality_result
    pred_img = draw_bbox(
  File "/home/ava/anaconda3/envs/selventh/lib/python3.8/site-packages/mmdet3d/core/visualizer/image_vis.py", line 124, in draw_lidar_bbox3d_on_img
    return plot_rect3d_on_img(img, num_bbox, imgfov_pts_2d, color, thickness)
  File "/home/ava/anaconda3/envs/selventh/lib/python3.8/site-packages/mmdet3d/core/visualizer/image_vis.py", line 82, in plot_rect3d_on_img
    cv2.line(img, (corners[start, 0], corners[start, 1]),
cv2.error: OpenCV(4.9.0) :-1: error: (-5:Bad argument) in function 'line'
> Overload resolution failed:
>  - Can't parse 'pt1'. Sequence item with index 0 has a wrong type
>  - Can't parse 'pt1'. Sequence item with index 0 has a wrong type

junjie18 / CMT

nuscenes 3D框投影到图片上可视化的问题 #96