search

nianticlabs / marepo

[CVPR 2024 Highlight] Map-Relative Pose Regression for Visual Re-Localization
https://nianticlabs.github.io/marepo/
Other
104 stars 8 forks source link

Unable to reproduce results in Table 1 #4

Closed MubarizZaffar closed 2 months ago

MubarizZaffar commented 2 months ago

Hey guys, fantastic work! Very easy to set up the repo too, thanks for open-sourcing as always.

With the released and downloadable models (both ACE and marepo), we should be able to reproduce the results in Table 1 of the paper? I quickly copy your Table below for ease of reference.

image

I am getting a median translation error of 6.66 cm (instead of the 3.9 cm reported in the paper) and a median rotation error of 1.84 degrees on the 7-scenes dataset. I have posted the terminal snapshot below, just to make sure that I am looking at the right printout. I followed all the steps in your readme as-is. I did not modify any of the params in test_7scenes.sh and I am not training/fine-tuning any of the models on my end. Many thanks.

image

tcavallari commented 2 months ago

Hi, thank you for reporting the issue.

What you see in the screenshot you attached is the median error for the last sequence of the 7Scenes dataset (Stairs), and it indeed matches the table in the paper.

There was a bug in the public version of the code causing it to not print the average median error across the dataset. That has been fixed in this commit: https://github.com/nianticlabs/marepo/commit/9a45e2bb07e5bb8cb997620088d352b439b13e0e

Can you please pull and retry the evaluation script? It should work correctly now.

MubarizZaffar commented 2 months ago

Yeap, it's resolved now. Thanks!