Weird & inconsistent performance in KITTI dataset

[LimHyungTae]

Hello!

I'm testing the robustness of the MDGAT-matcher when two viewpoints of the point cloud is quite distant.

But, there is a weird and inconsistent performance.

In seq. 00, 05, 06, the performance is quite reasonable, yet in seq 02, MDGAT-matcher just failed, and in 07 and 08, performance whose viewpoint difference ranges between 6 and 10m showed better performance than 2~6m.

Success rate criteria is changed into < 2m && < 10 deg.

Could you share your insight?

[LimHyungTae]

I also uploaded my preprocess files.

preprocess-.zip

[chenghao-shi]

Hi！

Thank you for expressing interest in our research. I have replicated the results that you mentioned above when using the preprocessed files you provided. I am curious if you have tested other methods on the same data and how they perform.

I observed that the data was generated on the closed loop. I suggest checking the ground truth poses calculated as I suspect that they may not be accurate. In fact, I have included three visualized examples of registered point clouds using your provided poses that reflect this issue. To address this, I recommend refining the ground truth pose using an ICP. Alternatively, you could use the refined poses provided by semantic KITTI (belowing), which have undergone loop correction. semantic-kitti-poses.zip

The performance degradation of current model on seq. 02 and 08 is the predictable, as these sequences contain reverse loops and the pretrained model was trained without data augmentation. If you want to generate better results, I believe applying data augmentation can help.

[LimHyungTae]

Really appreciate your prompt reply! Yet, I already used semantic-Kitti-poses as ground truth.

I thoroughly checked your approach's results, and I saw that in some cases, your approach returned all the minus ones (i.e., -1) of matches, resulting in empty mkpts0, whose size is 0 x 3.

The corresponding part is as follows: [in test_registration_metric.py]

matches, matches1, conf = pred['matches0'][b].cpu().detach().numpy(), pred['matches1'][b].cpu().detach().numpy(), pred['matching_scores0'][b].cpu().detach().numpy()
valid = matches > -1

mkpts0 = kpts0[valid]
mkpts1 = kpts1[matches[valid]]
mconf = conf[valid]

Thus, the empty mkpts0 and mkpts1 triggers breakdown of the evaluation pipeline.

As you mentioned earlier, the ground truth of seq 02 and 08 is quite imprecise; nevertheless, I guess your approach doesn't work in the reverse cases, failing to estimate correspondences (no offense! I'd just share my observations).

Please visualize the estimated results, or I'd like to know the information of preprocess-undownsample-n8 point clouds. With no information, I cannot visualize and qualitatively analyze the results :(

The commented parts:

# pc0_path = os.path.join('/home/chenghao/Mount/Dataset/KITTI_odometry/preprocess-undownsample-n8', pred['sequence'][b], '%06d.bin'%pred['idx0'][b])
# pc1_path = os.path.join('/home/chenghao/Mount/Dataset/KITTI_odometry/preprocess-undownsample-n8', pred['sequence'][b], '%06d.bin'%pred['idx1'][b])
# pc0, pc1 = np.fromfile(pc0_path, dtype=np.float32), np.fromfile(pc1_path, dtype=np.float32)
# pc0, pc1 = pc0.reshape(-1, 8), pc1.reshape(-1, 8)

[chenghao-shi]

Ya, MDGAT is not designed to be rotational invariant and currently is only trained using continuous pairs without yaw augmentation, so it can not handle reverse loops yet.

You can just ignore 'preprocess-undownsample-n8', and read your raw point cloud in pc0 and pc1 should work. You may need to add if mkpts0.shape[0]>0: before line281 as there may not be match.

[LimHyungTae]

Or this would be better:

if opt.calculate_pose:
    if (mkpts0.shape[0] == 0 or mkpts1.shape[0] == 0):
       T = torch.tensor(np.eye(4))
       T_error = pred['T_gt'][b].cpu().double().numpy()
       rte = np.linalg.norm(T_error[:3, 3])
       f_theta = (T_error[0, 0] + T_error[1, 1] + T_error[2, 2] - 1) / 2
       rre = np.arccos(f_theta)
    else:
        T, rte, rre=\
        calculate_error2(mkpts0, mkpts1, b, pred['T_gt'][b])

which means that if the registration fails, then the GT rel pose is just considered as an error.

[LimHyungTae]

All issues are resolved. Thank you for helping me!