Reproducing Paper Results

[yuvalH9]

Hi @liuQuan98 ,

Thank you very much for your interesting work and your detailed code.

I was trying to reproduce the results shown in the paper on kitti (table 1 and table 3), using your weights and the supplied test script.

I am not sure which flag I should use to reproduce the Registration Recall of <1.5 <60.

I saw that when using --lokitti false the test size is 225 (with maximal gap between frame of ~35m).

When --lokitti true we have ~550 problems.

Should I use --useRansac false as well?

Can you guide me how to reproduce paper results?

Thanks Yuval

[liuQuan98]

Hi @yuvalH9 ,

Thank you for your interest in our work. Setting the registration thresholds at 1.5 degrees and 60cm requires you to set rre_thresh and rte_thresh to 1.5 and 0.6, respectively.

Parameter 'lokitti' is another benchmark released in my previous publication on this topic (see my liuQuan98/APR for details). It selects pairs with only <=30% overlap, mimicking the 3DLoMatch proposed by OverlapPredator. For more details on the setup of this dataset, please refer to this paper.

Parameter 'use_RANSAC' should always be true, so that we are conducting fair comparison using RANSAC with other methods. Only when you are not interested in comparing performance, you can disable it to use a faster alternative (which is actually SC2-PCR) to generate better results at a higher throughput.

The problem size for lokitti is fixed, and when lokitti is enabled, it overrides pair_min_dist and pair_max_dist. To produce my results in Table1, run test_gcl_kitti.sh script with lokitti false rre_thresh 1.5 rte_thresh 0.6 pair_min_dist X pair_max_dist Y where (X,Y) is one of the following: (5,10), (10,20), (20,30), (30,40), (40,50). These five experiments represent results on five test dataset splits in Table 1. As for Table 3, I didn't release the model other than the best one. You can check its performance by setting lokitti true for the lokitti results, and setting lokitti false pair_min_dist 10 pair_max_dist 10 for the KITTI[10,10] results.

Best, Quan Liu