Using models in mmdetection and mmdetection3D

[ZiadElmassik]

Hello Mathew, I have converted the CADCD to KITTI format. In order to test 3D and 2D detection models in mmdetection2D and 3D do I need to create config and checkpoint files tailored to the CADCD or can I just use the ones made for the KITTI dataset? If I have to create config files can you give me an idea as to how I could do this? I'm currently trying to test the models SECOND and MVX-Net in mmdetection 3D.

[rvalienter90]

Hello @Balazizo I am also working on converting CADCD to KITTI, but I am still getting problems with the calibration files, I could be missing something, but is not clear to me how can I get R0_rect, T_cam_to_velo, P0, and D0 from the current calib? as in kitti format. I am confused with that conversion. I using the code from https://github.com/mathild7/faster_rcnn_pytorch_multimodal, but calib does not save R0_rect, T_cam_to_velo, P0 and D0. I really appreciate your help on that. Thanks!!!

def make_calib_file(calib, calib_target_path):
...

https://github.com/mathild7/faster_rcnn_pytorch_multimodal/blob/3145d91cba5ec38769b76af8f6e118748876c280/tools/cadc_unpack_all_kitti.py#L76

[ZiadElmassik]

Hello @rvalienter90 I'm afraid I'm not sure as to how to convert the calib files yet. I'm currently only interested in the annotations as I'm trying to evaluate object detection algorithms on the dataset.

[ZiadElmassik]

Hello @rvalienter90, I've looked into the conversion of the CADCD calibration files to the KITTI format and I think I see the relation a bit better. In the first two screenshots, an example of original CADCD calibration .txt files is in the left text editor and in the right is an example of the calib file in the KITTI 3D detection dataset. In the 3rd screenshot, I removed some of the unwanted matrices since I'm only working with CAM00 and the lidar. If your case is similar then what I'm about to say should apply to you as well. CAM00 in the CADCD is the equivalent of Camera 2 in the KITTI dataset. However, since the cameras in the KITTI dataset have undergone rectification, instead of using K_02 for camera 2, they used P2 which is the projection matrix post-rectification. To produce the P for CAM00 in the CADCD is very simple; in our case the rectification matrix is a 3x3 identity matrix meaning the projection matrix (P) is literally just the CAM00_matrix (which can be represented by K) but 3x4 (CAM00 matrix but add a 3x1 column from the right of all 0s and 4x1 row from underneath of (0 0 1 0) ). This new matrix will take the place of CAM00_matrix as let's say P_00.

R0_rect is just a 3x3 identity matrix since I'm working with only 1 camera.

T_LIDAR_CAM00 in CADCD is literally the same as Tr_velo_to_cam but in homogenous coordinates so just remove the 0 0 1 at the end of T_LIDAR_CAM00.

Now all that's left is CAM00_distortion: This part is slightly more tricky since the KITTI 3D detection dataset is undistorted so we will have to modify the script in order to undistort the camera image using cv2. undistort(----). After that, we can remove CAM00_distortion from the calibration files.

These are the modifIcations that should be made if you're working with CAM00 and the lidar only. I'm currently working on completing these steps. Once they're done, the CADCD should look exactly like the KITTI dataset.

@mpitropov, anything wrong with these steps?

[rvalienter90]

Hello @Balazizo Thank you very much, I agree with you. I also checked the calibration files and I believe that we mainly need P2, R0_rect, and Tr_velo_to_cam. I agree that R0_rect is the identity matrix. Not sure if Tr_velo_to_cam should be exactly T_LIDAR_CAM00, it made needs some reference change, is not clear what is T_LIDAR_CAM00 for me. But yes for now I am using same removing the last values. I am also not sure about camera distortion as is not need it in KITTI.

@mpitropov can give us some insights on that. Please let us know if we are in the right path.

I created this script based on that, but the values are not correct , the 3D projections on camera images are wrong.

def make_calib_file(calib, calib_target_path):

    T_LIDAR_CAM = (np.array(calib['extrinsics']['T_LIDAR_CAM0' + cam]));
    T_CAM_LIDAR = np.linalg.inv(T_LIDAR_CAM)

    T_IMG_CAM = np.eye(4);
    T_IMG_CAM[0:3, 0:3] = np.array(calib['CAM0' + cam]['camera_matrix']['data']).reshape(-1, 3);
    T_IMG_CAM = T_IMG_CAM[0:3, 0:4];  # remove last row
    # T_CAM_IMG =np.linalg.inv(T_IMG_CAM)
    dist_coeffs = np.array(calib['CAM0' + cam]['distortion_coefficients']['data'])

    R = 'R0_rect: ' + ' '.join([str(x) for x in np.eye(3).reshape(-1).tolist()])
    velodyne = T_CAM_LIDAR[0:3, 0:4];
    velodyne = 'Tr_velo_to_cam: ' + ' '.join([str(x) for x in velodyne.reshape(-1).tolist()])

    P = T_IMG_CAM
    P = ' '.join([str(x) for x in P.reshape(-1).tolist()])

    info = f"P0: {P}\nP1: {P}\nP2: {P}\nP3: {P}\n{R}\n{velodyne}\n"
    f_c = open(calib_target_path, "w+")
    f_c.write(info)

    return info

The calb files output looks like this now

P0: 653.956033188809 -0.235925653043616 653.221172545916 0.0 0.0 655.54008861796 508.732863993917 0.0 0.0 0.0 1.0 0.0
P1: 653.956033188809 -0.235925653043616 653.221172545916 0.0 0.0 655.54008861796 508.732863993917 0.0 0.0 0.0 1.0 0.0
P2: 653.956033188809 -0.235925653043616 653.221172545916 0.0 0.0 655.54008861796 508.732863993917 0.0 0.0 0.0 1.0 0.0
P3: 653.956033188809 -0.235925653043616 653.221172545916 0.0 0.0 655.54008861796 508.732863993917 0.0 0.0 0.0 1.0 0.0
R0_rect: 1.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 1.0
Tr_velo_to_cam: 0.01394524701841737 -0.9998000763361616 -0.01514716891344736 -0.06268651366968261 -0.005120798147914583 0.015064872783513402 -0.9998855747124629 -0.6056567052837293 0.9999020731035041 0.01403306014823503 -0.004921372004126455 -0.6618044419813216

The format is correct but the values are not right. I am also working on that, if I have need progress I will share it. @Balazizo Please let me know if you able to fix the camera part so calib files from CADCD looks exactly like KITTI. Having the CADCD datasets in KITTI format will be very helpful for a lot of people. Thanks!

[ZiadElmassik]

Hello @rvalienter90, I'm currently working on undistorting the image within the conversion script and writing the undistorted in the place of the distorted one. I'm dealing with a few bugs at the moment. Once, I output a result, I'll put the modified script here. I'm working with the cv2.undistort() function to do this. You can try looking into that.

[mpitropov]

There are too many topics being discussed in this issue.

I have not used either mmdetection and mmdetection3D but I have used OpenPCDet. I got CADC working on it by creating a custom data loader for it.

The annotated dataset comes undistorted already. I think that the issues with the calibration conversion would be easier if you looked into my own example scripts. For example, https://github.com/mpitropov/cadc_devkit/blob/master/run_demo_tracklets.py does not use the distortion values because all the camera images that are annotated are given undistorted. On the other hand, https://github.com/mpitropov/cadc_devkit/blob/master/run_demo_lidar.py can be run on both the raw and processed data. Due to this, it will use the distortion coefficients on the raw data https://github.com/mpitropov/cadc_devkit/blob/9c43a311ad6c9262fa2e0ce8bc60638c2ab4dda9/lidar_utils.py#L90

[ZiadElmassik]

Oh, I see. Thanks, @mpitropov. So, since all the images are labelled(at least in my download), none of them are distorted. Hence, the distortion coefficients are not needed?

[mpitropov]

Yes, you don't need the distortion coefficients. Adding them shouldn't hurt anything though.

[ZiadElmassik]

Alright, thank you Mathew. May I ask something, though? When I was testing the cv2 functions getOptimalMatrix() and undistort(), they returned a slightly different intrinsic matrix and after viewing the resulting image, it was very similar except the image was slightly cropped from each side. Should I keep these changes or just remove them?

[mpitropov]

Have you looked at and compared the raw and processed datasets? It sounds like you are running undistort on images that I already undistorted. Can you add a comparison?

An example from 2018_03_06 sequence 0001. The left image is raw while the one on the right is processed (undistorted):

[ZiadElmassik]

Hi @mpitropov, I've run the undistortion and optimalCameraMatrix on image 00000 within the labelled folder in sequence 0001, drive 2018_03_06. I ended up with the result on the right. The one on the right seems slightly cropped from the left, down, right and top.

[MartinHahner]

Interesting discussion going on here. 🙂 @mpitropov: I will soon start using your OpenPCDet loader. I hope it works "out-of-the-box" as you say. 😄

Hi @mpitropov, I've run the undistortion and optimalCameraMatrix on image 00000 within the labeled folder in sequence 0001, drive 2018_03_06. I ended up with the result on the right. The one on the right seems slightly cropped from the left, down, right and top.

@Balazizo: The image on the right looks totally distorted (even though you say you ran some undistortion on it). Just look at the white lane markings in front of the car, they are not straight at all (as they should be in an undistorted image).

[ZiadElmassik]

Good catch @MartinHahner. I probably won't fiddle around with undistortion given the images already undistorted.