Poor test results from one checkpoint

[zjx99]

Hi, Weijing, I used the checkpoint car_auto_T3_trainval to do initial test on KITTI test dataset with the following command: python3 run.py checkpoints/car_auto_T3_trainval/ --test --dataset_root_dir DATASET_ROOT_DIR --output_dir DIR_TO_SAVE_RESULTS. However, the results I got are too bad because only a few cars can be detected. Also, it is strange that all the detected bounding boxes are only located on the right side of the frames, shown as figures below. Do I need to change any parameters？ Could you please give me some advice? Thanks.

[WeijingShi]

Hi @zjx99 ,

The KITTI detection benchmark only provides annotations that are visible in the image plane. The training data and labels will probably give your similar visualization results. Point-GNN is trained to detect objects within the same area. In fact, we remove points outside the image during preprocessing. To handle all the objects with 360 degrees, ideally, we would want to get all the annotations and retrain the network. But a quick hack you can pull is to rotate the point cloud to multiple degrees and combine the detection results. Hope it helps.

[zjx99]

Thank you very much! It is a smart idea to rotate the point cloud in order to get all the detections. I will try it and hope better results will come out.

[zjx99]

Hi, Weijing, I am trying to do the test on my own lidar data. But the problem is that I don't have images in accordance with lidar, so the obtained txt files are all empty when using the default KITTI images since they don't match with my lidar data. I tried to remove the restriction of the images in kitti_dataset.py, but got many errors in the end. Any advice on this issue? Thanks.

[WeijingShi]

Hi @zjx99, I am not sure about your sensor configuration. In general, if you want to use the checkpoint pretrained on KITTI, you need to do a few conversions. First, convert your point cloud to the KITTI camera coordinates. You only need to align the axis through rotation, Point-GNN is invariant to global translation. Save the converted points.

In the run.py, replace dataset.get_cam_points_in_image_with_rgb() with a method to load your point cloud. something like get_velo_points(). Also, you might want to crop your point cloud to the camera front area to save some GPU memory. https://github.com/WeijingShi/Point-GNN/blob/48f3d79d5b101d3a4b8439ba74c92fcad4f7cab0/dataset/kitti_dataset.py#L587

The visualization won't work as it uses RGB. But the script should be able to process your point cloud.

[zjx99]

Thanks a lot for the solution. However, my target is to detect all the vehicles using the pre-trained model and show the boxes just in the point cloud, which makes images not useful to me at all. Since the images are only used for visualization in the whole process of your work, could I just directly remove them and do the test on the original point cloud? Is there any particular reason that I have to convert my point cloud to the KITTI camera coordinate system?

[WeijingShi]

Hi @zjx99,

The reason we need to convert your point cloud to the KITTI camera coordinates is that the checkpoint is trained in KITTI camera coordinates. e.g., in KITTI the y-plane is the ground plane, etc. If your point cloud has z-plane as the groud, the checkpoint won't understand that. It's a pre-processing step to match the custom dataset with the training data. You can then convert the detection results back to your coordinate system.

[zjx99]

Hi, Weijing, I've converted my point cloud to the coordinate system of KITTI test data by registration( shown in the first figure, red is mine), and then load it through get_velo_points function, which has been revised as shown in the second figure to accept csv file. However, the problem is that the txt file after detection are still empty. I didn't transfer my lidar to KITTI camera coordinates directly as you suggested because I think as long as my lidar has been aligned with the KITTI lidar data, the imported points will be transferred later by the get_cam_points function. Is there any problem?

Moreover, I don't really understand the difference of the following two functions, could you please give more explanation?

Thank you very much

[WeijingShi]

Hi @zjx99,

Can you help to explain your "registration" process? After "registration", does the point cloud have the coordinate axis like the followed?

In the run.py, do you use get_cam_points() instead of get_cam_points_in_image()?.

You can use the visualization without rgb by removing the following line. This might help for debugging.
https://github.com/WeijingShi/Point-GNN/blob/48f3d79d5b101d3a4b8439ba74c92fcad4f7cab0/run.py#L161

velo_points_to_cam() and velo_to_cam() is doing the same thing with different input type. points are a namedtuple, points_xyz are [N, 3] np.array.

[zjx99]

Hi, Weijing, I used ICP to do the registration, in which the reference is the KITTI point cloud(R), and the template is my point cloud(T). After registration , ideally, T has been converted to the coordinate system of R, which means the axes are parallel and the origins are almost the same( without considering the translation currently). And also, I used get_cam_points() in run.py. Unfortunately, there were still no results coming out.

[WeijingShi]

Hi @zjx99,

From the image, it seems you ran ICP between the KITTI and your custom dataset. Since they are not from the same scene, the registration quality is poor. Is it ok to manually check if they have a similar coordinate system after R? Particularly, do they have the same y-axis (height)?

[zjx99]

Hi, Weijing, I manually checked in Cloudcompare, they seem to be in similar system especially in terms of height. Also, to clarify, currently z-axis is the height because I was trying to convert my point cloud to the KITTI velo system in the first place, then it can be converted to the KITTI camera system by the calib file. It is the right procedure, isn't it? Also, now I have difficulty determining the coordinate system of my point cloud. I can't decide the directions of x-axis and y-axis, so actually I don't quite know how to convert it to KITTI system without considering registration. Any advice from you about deciding the system of my point cloud? Thanks.

[WeijingShi]

Hi @zjx99, sorry for the late reply. Yes, you are right, the z-axis should be the height in the lidar frame. We can try to decide the axis by manually find some vertical objects such as a wall or lamp pole and set the z-axis. However, it's unclear how sensitive is the network to the calibration error. If you have annotations in your dataset, maybe you can retrain or finetune the checkpoints. Thanks.

[GXQ0527]

@zjx99 您好！想问一下，换成自己采集的激光雷达数据集您这边现在能够检测吗？

[zjx99]

@GXQ0527 , Hi, No, I didn't get good results, so I changed to other methods.