search

open-mmlab / mmdetection3d

OpenMMLab's next-generation platform for general 3D object detection.
https://mmdetection3d.readthedocs.io/en/latest/
Apache License 2.0
5.18k stars 1.52k forks source link

Is the calib data also used durinig 3d box-related training? ( kitti ) #1261

Open ekzmzm79-github opened 2 years ago

ekzmzm79-github commented 2 years ago

hello

I want to create a 3d-bbox inference model using only 3d-lidar pcl data. Also, I confirmed that the data is created as shown below when I run the 'create_data.py' code through kitti data. image

From here on, I decided that calib and image data were not needed. because ground-truth files included in 3d-bbox have already been created.

Car AP@0.70, 0.70, 0.70: bbox AP:0.0000, 0.1057, 0.1057 bev AP:0.0000, 0.0000, 0.0000 3d AP:0.0000, 0.0000, 0.0000 aos AP:0.00, 0.01, 0.01 Car AP@0.70, 0.50, 0.50: bbox AP:0.0000, 0.1057, 0.1057 bev AP:0.0000, 0.1057, 0.1057 3d AP:0.0000, 0.1057, 0.1057 aos AP:0.00, 0.01, 0.01

I thought only 3d ap would measure properly, but as you can see, training doesn't seem to work well. 20220222184055.log

The code I have modified is as follows, but the purpose is to replace only calib and image information in the info pkl file with a dummy after data is created in create_data.py.

import argparse
from distutils.log import info
from os import path as osp

from pandas import array

from tools.data_converter import indoor_converter as indoor
from tools.data_converter import kitti_converter as kitti
from tools.data_converter import lyft_converter as lyft_converter
from tools.data_converter import nuscenes_converter as nuscenes_converter
from tools.data_converter.create_gt_database import create_groundtruth_database

import mmcv

def kitti_data_prep(root_path, info_prefix, version, out_dir):
    """Prepare data related to Kitti dataset.

    Related data consists of '.pkl' files recording basic infos,
    2D annotations and groundtruth database.

    Args:
        root_path (str): Path of dataset root.
        info_prefix (str): The prefix of info filenames.
        version (str): Dataset version.
        out_dir (str): Output directory of the groundtruth database info.
    """
    kitti.create_kitti_info_file(root_path, info_prefix)
    kitti.create_reduced_point_cloud(root_path, info_prefix)

    info_train_path = osp.join(root_path, f'{info_prefix}_infos_train.pkl')
    info_val_path = osp.join(root_path, f'{info_prefix}_infos_val.pkl')
    info_trainval_path = osp.join(root_path,
                                  f'{info_prefix}_infos_trainval.pkl')
    info_test_path = osp.join(root_path, f'{info_prefix}_infos_test.pkl')

    create_groundtruth_database(
        'KittiDataset',
        root_path,
        info_prefix,
        f'{out_dir}/{info_prefix}_infos_train.pkl',
        relative_path=False,
        mask_anno_path='instances_train.json',
        with_mask=(version == 'mask'))

    for path in [info_train_path, info_val_path, info_trainval_path, info_test_path]:
        clear_calib_imageInfo(path)

    kitti.export_2d_annotation(root_path, info_train_path)
    kitti.export_2d_annotation(root_path, info_val_path)
    kitti.export_2d_annotation(root_path, info_trainval_path)
    kitti.export_2d_annotation(root_path, info_test_path)

def clear_calib_imageInfo(path):
    infos = mmcv.load(path)

    import numpy as np

    for obj in infos:
        obj["image"]["image_idx"] = 0
        obj["image"]["image_path"] = 'training/image_2/000000.png'
        obj["image"]["image_shape"] = np.array([ 375, 1242])

        obj["calib"]["P0"] = np.array([[721.5377,   0.    , 609.5593,   0.    ],
       [  0.    , 721.5377, 172.854 ,   0.    ],
       [  0.    ,   0.    ,   1.    ,   0.    ],
       [  0.    ,   0.    ,   0.    ,   1.    ]])
        obj["calib"]["P1"] = np.array([[ 721.5377,    0.    ,  609.5593, -387.5744],
       [   0.    ,  721.5377,  172.854 ,    0.    ],
       [   0.    ,    0.    ,    1.    ,    0.    ],
       [   0.    ,    0.    ,    0.    ,    1.    ]])
        obj["calib"]["P2"] = np.array([[7.215377e+02, 0.000000e+00, 6.095593e+02, 4.485728e+01],
       [0.000000e+00, 7.215377e+02, 1.728540e+02, 2.163791e-01],
       [0.000000e+00, 0.000000e+00, 1.000000e+00, 2.745884e-03],
       [0.000000e+00, 0.000000e+00, 0.000000e+00, 1.000000e+00]])
        obj["calib"]["P3"] = np.array([[ 7.215377e+02,  0.000000e+00,  6.095593e+02, -3.395242e+02],
       [ 0.000000e+00,  7.215377e+02,  1.728540e+02,  2.199936e+00],
       [ 0.000000e+00,  0.000000e+00,  1.000000e+00,  2.729905e-03],
       [ 0.000000e+00,  0.000000e+00,  0.000000e+00,  1.000000e+00]])
        obj["calib"]["R0_rect"] = np.array([[ 0.9999239 ,  0.00983776, -0.00744505,  0.        ],
       [-0.0098698 ,  0.9999421 , -0.00427846,  0.        ],
       [ 0.00740253,  0.00435161,  0.9999631 ,  0.        ],
       [ 0.        ,  0.        ,  0.        ,  1.        ]])
        obj["calib"]["Tr_velo_to_cam"] = np.array([[ 7.533745e-03, -9.999714e-01, -6.166020e-04, -4.069766e-03],
       [ 1.480249e-02,  7.280733e-04, -9.998902e-01, -7.631618e-02],
       [ 9.998621e-01,  7.523790e-03,  1.480755e-02, -2.717806e-01],
       [ 0.000000e+00,  0.000000e+00,  0.000000e+00,  1.000000e+00]])
        obj["calib"]["Tr_imu_to_velo"] = np.array([[ 9.999976e-01,  7.553071e-04, -2.035826e-03, -8.086759e-01],
       [-7.854027e-04,  9.998898e-01, -1.482298e-02,  3.195559e-01],
       [ 2.024406e-03,  1.482454e-02,  9.998881e-01, -7.997231e-01],
       [ 0.000000e+00,  0.000000e+00,  0.000000e+00,  1.000000e+00]])

    mmcv.dump(infos, path)

No modifications have been made to the "kitti_dataset" related and other codes yet. Can you please point out if I'm misunderstanding something?

If I'm misunderstanding, is there any guide on how to proceed with training with only 3d-lidar pcl information? I am debating whether to modify the kitti_dataset code or configure a completely new dataset.

i'm already read this https://github.com/open-mmlab/mmdetection3d/issues/260, https://github.com/open-mmlab/mmdetection3d/issues/334, https://github.com/open-mmlab/mmdetection3d/issues/429 . https://github.com/open-mmlab/mmdetection3d/issues/1173

thank you

Tai-Wang commented 2 years ago

The calibration can be used when pre-processing point cloud data. It will filter points outside of the front view on KITTI. As for a systematic solution, we are preparing a tutorial or doc tailored to this setting. Please stay tuned.

fathyshalaby commented 2 years ago

Is there an update reagarding the tutorial for only pcd based training and inference