Mismatch between accumulated point clouds and images

[isunLt]

I'm trying to do cross-modal training between lidar and camera with this dataset. Therefore I project the labeled accumulated point clouds to the images and cut out the points that are out of the fov of the camera to make the point cloud correspond to the image. However, I find the accumulated point cloud may contain fake points that do not correspond to the real scenes captured by the image. What might cause this problem?

For example, sequence: 2013_05_28_drive_0000_sync, frame id: 250

Fig 1. Accumulated point cloud is projected to the images. In the red circle, many green points(stand for vegetation) are projected to the wall.

Fig 2. Raw velodyne point cloud is projected to the images. In the red circle, many grey points(stand for wall) are projected to the wall which means the raw point cloud and images are correspond.

Fig 3. The accumulated point cloud(colorful points) and raw velodyne point cloud(red points) are project to the same coordinate system(the world coordinate). I find in the blue circle there are redundant green points which cause the mismatch problem. But where do they come from?

[yiyiliao]

Hi, thank you for your question. One possible reason for the fake points might be the spurious estimation of stereo matching. Our accumulated point cloud is composed of observations of three sensor modalities: Velodyne laser scanner, SICK laser scanner and stereo camera:

The stereo estimation should be reliable in most cases though, as we performed left-right consistency check and multi-view consistency check across neighboring frames. We also ignored points further than 15 meters in each frame.

Can you let me know the frame ID? I can check the stereo depth estimation of the neighboring frames to verify if this is the cause.

[isunLt]

Hi, thank you for your question. One possible reason for the fake points might be the spurious estimation of stereo matching. Our accumulated point cloud is composed of observations of three sensor modalities: Velodyne laser scanner, SICK laser scanner and stereo camera:

The stereo estimation should be reliable in most cases though, as we performed left-right consistency check and multi-view consistency check across neighboring frames. We also ignored points further than 15 meters in each frame.

Can you let me know the frame ID? I can check the stereo depth estimation of the neighboring frames to verify if this is the cause.

Hi, thanks for your quick respondence. The frame id is 250 in sequences 2013_05_28_drive_0000_sync.

[eslambakr]

Hello @isunLt

I am using also the accumulated point-clouds alongside the RGB frames but I faced another issue actually which is some files are empty. For example : 1) KITTI-360/data_3d_semantics/train/2013_05_28_drive_0000_sync/dynamic/0000000002_0000000385.ply 2) KITTI-360/data_3d_semantics/train/2013_05_28_drive_0000_sync/dynamic/0000000372_0000000610.ply 3) KITTI-360/data_3d_semantics/train/2013_05_28_drive_0000_sync/dynamic/0000000599_0000000846.ply

All of them has size of 311bytes. I downloaded the file several times and each time the files are empty like that. Did u faced the same issue?

Thanks in advance.

[isunLt]

Hello @isunLt

I am using also the accumulated point-clouds alongside the RGB frames but I faced another issue actually which is some files are empty. For example :

1. KITTI-360/data_3d_semantics/train/2013_05_28_drive_0000_sync/dynamic/0000000002_0000000385.ply
2. KITTI-360/data_3d_semantics/train/2013_05_28_drive_0000_sync/dynamic/0000000372_0000000610.ply
3. KITTI-360/data_3d_semantics/train/2013_05_28_drive_0000_sync/dynamic/0000000599_0000000846.ply

All of them has size of 311bytes. I downloaded the file several times and each time the files are empty like that. Did u faced the same issue?

Thanks in advance.

I do find these files are empty. As for me, I mainly focus on files in static folders. So it does not bother me.

[xuqinwang]

Thank you for your great work. Could you please tell me that whether there is label to annotate the sensor resource in your provided accumulated point cloud (velodyne, stereo or SICK), exp. data_3d_semantics/train/2013_05_28_drive_0000_sync/static/0000000002_0000000385.ply file? Many thanks!