mp3d-loftr, evaluation - Githubissues

SongMingtao-NUDT commented 2 months ago

Hello, I have some question about the Matterport-3D evaluation: Because the size of "cached_set_test" is too large(about 7996 pairs of images) for my compute, so I extracted a smaller dataset (about 135 pairs of images). Then I used it do the evaluation two times, but I got two different results. Here is one of the results: tr rot mean err 4.65 tr rot median err 0.35 tr rot pct < 30 94.8 tr abs mean err 0.42 tr abs median err 0.18 tr abs pct < 1 93.3 rot mean err 5.29 rot median err 0.23 rot pct < 30 95.6 pct successful fits 0.0 dset size 135

And here is another: tr rot mean err 4.09 tr rot median err 0.36 tr rot pct < 30 96.3 tr abs mean err 0.42 tr abs median err 0.17 tr abs pct < 1 93.3 rot mean err 5.22 rot median err 0.2 rot pct < 30 95.6 pct successful fits 0.0 dset size 135

Same weights, same inputs, I think we should get the same results. What do you think caused this situation?

crockwell commented 2 months ago

Hi,

I believe you found "the result of the re-running evaluation is very close to the data in the paper" in [this issue](https://github.com/crockwell/far/issues/7)? Presumably that finding was on the full set? It could be a subset has higher variance if there is randomness. This would probably be during RANSAC, particularly for the prior-guided solver.

Hope this helps! Chris

SongMingtao-NUDT commented 2 months ago

Hi,

I believe you found "the result of the re-running evaluation is very close to the data in the paper" in [this issue](https://github.com/crockwell/far/issues/7)? Presumably that finding was on the full set? It could be a subset has higher variance if there is randomness. This would probably be during RANSAC, particularly for the prior-guided solver.

Hope this helps! Chris

Thank you for your explanation.

crockwell / far

mp3d-loftr, evaluation #9