Replicating results

[mattiadurso]

Following the discussions in #74 and #67, I ran COLMAP and GLOMAP on the (distorted) ETH3D MVS (DSLR) dataset. However, I’m struggling to reproduce your results and am encountering two main issues:

1. High variability between different runs (AUC scores can be from 0 or 90).
2. Consistently obtaining AUC scores around 20 points lower than your reported average.

Since I’m using the AUC function directly from PxSfm (as mentioned in #67), I’m wondering if you tested the models with any non-default parameters. Some potential factors I’ve considered include:

• Random seed (for feature extraction, matching, and mapping)
• Image reconstruction seed ID
• Whether intrinsics were passed manually
• (BA) intrinsics refinement

Any insight or advice on how to address these issues would be greatly appreciated.

Thank you for your time.

[lpanaf]

Hi, regarding the experiments, we have fed in the ground truth intrinsic, so that intrinsics are properly shared among images, and are initialized with nice values. Also, we use undistorted images, instead of raw ones. Have you obtained reasonable values for COLMAP as a starting point?

[mattiadurso]

we have fed in the ground truth intrinsic, [...] we use undistorted images

That was the point! Thanks. Now I get mostly the same scores as in the paper.

It's interesting that in some scenes I get better results (for both models) than the GLOMAP paper. Also, at least in one run I did, GLOMAP does not fail on exhibition hall and scores 84.6 AUC while COLMAP 84.5 (starting from the same database). Also old computer scores better on my run of GLOMAP while the COLMAP run it's the same of yours.

If you are interest we can discuss about it.

Thanks again!

[ahojnnes]

@mattiadurso Great that you managed to replicate our experiments. Some smaller bugs have been fixed since submission of the paper and some components of the pipeline have been improved. So, it is expected that the latest commits in the main branch will produce slightly different (hopefully always better :-)) results than reported in the paper. The same goes for COLMAP.

[lpanaf]

Hi @mattiadurso, great to hear that you reproduce the result. Just an additional note, the major difference between the current version and the result in the paper should be the initialization of rotation averaging. Now, I enabled initialization from the maximum spanning tree, which solved the problem for exhibition_hall by chance. If you are interested, you may try disabling this function (there is a flag called skip_initialization in the options of rotation averaging, then the result should be more similar to what we have in the paper :)