Doppelgangers support

[awarebayes]

There is a repository, which aims to avoid incorrect image registration called 'Doppelgangers': link. Implementing it into the pipeline should be straightforward, since it is a binary classifier, also I am currently working on it.

My pipeline is the following:

1. netvlad
2. pairs from retrieval
3. feature extractor (superglue)
4. feature matcher (lightglue)
5. !!!! Doppelgangers for each feature pair generates probability that they are a correct match
6. for pairs with probability < thresh delete them from matches.hdf5
7. run sfm
8. ??? profit

This issue is for mainly tracking pull request, as well, as maybe some help I will need along the way

[sarlinpe]

Yes we already looked into this - it looks very promising. After discussing with the author @RuojinCai and experiments by @veichta, it looks like the model might not generalize well to types of matches (especially sparse matches) that are different than it was trained on (dense LoFTR). If someone can train a new model for DISK/SP+LightGlue (or, better, that generalizes to any sparse matches), then I'd be happy to integrate it (and add it to COLMAP down the road).

For the integration, I see two alternate designs:

1. Create a new step that takes feature and match files and outputs a new match file from which incorrect image pairs are removed.

   python -m hloc.remove_flase_pairs --features path/to/features.h5 --matches path/to/matches.h5 --output path/to/cleaned_matches.h5

2. Create a new step that adds a doppelganger_score attribute to each image pair group in the match file:

   python -m hloc.add_doppelganger_score --features path/to/features.h5 --matches path/to/matches.h5

   And skip importing pairs with doppelganger_score < threshold in hloc.triangulation.import_matches.

Design 2. makes it easier to run the reconstruction with different thresholds without running the doppelganger network again, but it is more invasive - so I'd go for 1. for now.

cc @Phil26AT

[awarebayes]

The doppelganger score is very weird, it's just logits of size (2,), and not probabilities. We can of course softmax them

[awarebayes]

Here is what I ended up doing: https://github.com/awarebayes/doppelgangers-hloc#usage-with-hloc

I need some more data from hloc, in my matches.h5, I created a merge request: https://github.com/cvg/Hierarchical-Localization/pull/309

I tested it with lightglue, somewhat works. Interesting remarks about superpoint+lightglue.

As for the final stage, I overwrite matches.h5 directly and delele 'bad' pairs: https://github.com/awarebayes/doppelgangers-hloc/blob/main/doppelgangers/utils/overwrite_hloc.py#L8

I believe it is possible to train doppelgangers since the architecture is a very simple CNN, and dataset is public and not really large.

[awarebayes]

And here is how I use HLOC matches and features to load the data: https://github.com/awarebayes/doppelgangers-hloc/blob/main/doppelgangers/datasets/hloc_dataset.py

[awarebayes]

I will try to train doppelgangers with superpoint + lightglue today