search

PRBonn / OverlapNet

OverlapNet - Loop Closing for 3D LiDAR-based SLAM (chen2020rss)
MIT License
662 stars 115 forks source link

Reproducing results of Fig.7 in paper #10

Closed fuying-wang closed 3 years ago

fuying-wang commented 3 years ago

Hi,

Thanks for your awesome work!

I am wondering how can I get the evaluation result of OverlapNet like Fig. 7? This is because I am planning to compare several other methods with OverlapNet, and they are using the top k recall and PR curve as metrics.

Chen-Xieyuanli commented 3 years ago

Hey @fuying-wang,

Fig. 7 is the evaluation results of the KITTI Odometry dataset sequence 00.

We followed the experimental setup of the OEROS paper. For the evaluation, point clouds from the first 170s of sequence 00 are used to generate the map, i.e., to populate the KD-tree. The remaining point clouds are used for localization queries. This split of sequence 00 prevents any self-localization, as the vehicle starts to revisit previously traversed areas after 170s. The query point clouds are sampled to be at least 3m apart.

In this specific experiment, we vary the number of top k nearest place candidates retrieved from the map and check whether the true positive is inside the candidates or not. The baseline results were directly taken from the OEROS paper.

I hope this helps!

fuying-wang commented 3 years ago

Hi,

Thanks very much for the details! That's very helpful.

fuying-wang commented 3 years ago

Hi, @Chen-Xieyuanli

I am still confused about two things. I am wondering if you can help me when you are free.

  1. In the OEROS paper, I saw:

    the query point clouds are sampled to be at least 3m apart.

I am not sure what this means. Does this mean I should down-sample the KITTI sequence 00 to ensure each two consecutive point clouds are 3m apart?

  1. I am wondering how I can get the true-positive samples. Maybe I can find all reference point clouds relative to the query point cloud within a certain distance, for example, 1.5m. Does this right? Also, are you consider the most similar one as the true-positive in this experiment? Or just all found point clouds within this distance?

Thanks a lot!

Chen-Xieyuanli commented 3 years ago

Does this mean I should down-sample the KITTI sequence 00 to ensure each two consecutive point clouds are 3m apart?

Yes.

Maybe I can find all reference point clouds relative to the query point cloud within a certain distance, for example, 1.5m. Does this right?

Yes.

Also, are you consider the most similar one as the true-positive in this experiment? Or just all found point clouds within this distance?

After downsampling, there should be only one true-positive, since candidates are 3m away from each other and only the one inside the 1.5m area is the true positive.

fuying-wang commented 3 years ago

Thanks very much!

Chen-Xieyuanli commented 3 years ago

Since there is no further question about this issue, I'm going to close it.

If there is any other problem, please feel free to ask me to reopen it.