search

cvg / Hierarchical-Localization

Visual localization made easy with hloc
Apache License 2.0
3.19k stars 594 forks source link

Cannot reproduce the benchmarks' result on InLoc dataset #82

Open liyichao108 opened 3 years ago

liyichao108 commented 3 years ago

Hi, @Skydes I tried running "pipeline_inloc.ipynb" with the parameters marked on the leaderboard, and using the "pairs-query-netvlad40-temporal.txt" as the input of matcher, but the results is not as good as the leaderboard. The parameters I use: SuperPoint features (4096 keypoints, NMS radius 3, on 1600-px images), and SuperGlue matcher (sinkhorn_iterations 50, outdoor model). Skip images with less than 15 matches. The results on InLoc dataset are as followed: duc1: 47.0 / 68.2 / 80.8; duc2: 49.6 / 77.1 / 81.7 While the result on leaderboard is: duc1: 49.0 / 68.7 / 80.8; duc2: 53.4 / 77.1 / 82.4

I would like to know if I'm using the right parameters, and is it convenient to describe the method of temporal consistency in image retrieval module in detail. Hope to hear from you soon. Thank you so much!

sarlinpe commented 3 years ago

The only source of randomness in this pipeline is RANSAC for the pose estimation. Do you get different results if you run this step a second time? The parameters should be correct.

Regarding the temporal consistency: I assume that the trajectory successively goes into the two floors. An initial pose estimation without temporal consistency provides an initial vote on the floor corresponding to each query. Assuming that the floor change happens a single time, i.e. for one query image, I then exhaustively compute consistency scores for all queries at each hypothesis and pick the most consistent.

liyichao108 commented 3 years ago

I run the pose estimation module second time, and the result did not change.