Robustness against rotation and flipping

[repoit]

For my use case, images might be flipped or rotated. However, the model doesn't perform well on these images (see the example below). Do you believe the model would get robust against these transformations if I added augmentation for rotation and flipping during the training process?

Two images showing the same scene with high matching rates

When flipped, no good matches are detected

[Varun-Tandon14]

@repoit you might want to try SE2-LoFTR (official repo). From my limited experience, SE2-LoFTR seemed to perform reasonably well for rotations of up to 30 degrees. Hopefully, this helps.

[zehongs]

Augmentation will be helpful. I believe SE2-LoFTER can also handle this problem to a certain extent. Another useful trick is to run LoFTR multiple times, e.g. [{image0, image1}, {rot90-image0, image1}, {rot180-image0, image1}, {rot270-image0, image1}, {flipped-images0, image1}] and take the union of all derived matches . While this can be batched, it can require a lot of GPU memory and a longer time😂

[repoit]

Augmentation will be helpful.

Great. I will then setup the training pipeline using MegaDepth and augment the existing training data with rotated/flipped cases. Is there anything important I should consider before doing so, or is it sufficient to just augment the data and follow your instructions for training?

Another useful trick is to run LoFTR multiple times, e.g. [{image0, image1}, {rot90-image0, image1}, {rot180-image0, image1}, {rot270-image0, image1}, {flipped-images0, image1}] and take the union of all derived matches . While this can be batched, it can require a lot of GPU memory and a longer time😂

I like this approach, but efficiency is key in our application.

SE2-LoFTER

The SE2 model does indeed work better (see example below). I believe it's interesting to compare the results using SE2 to LoFTR when trained with augmented data.

Comparison of SE2 non-flipped (left) and flipped (right) when using the 8rot.ckpt model weights

[zehongs]

Training LoFTR requires lots of GPUs. And augmentation is likely to hurt the performance of matching "normal" image pairs.

[ACSL-ricardo]

@repoit Did you or anyone finally apply data augmentation? It seems it is necessary to change the spvs_coarse function in supervision.py to be able to change the ground truth depending on the rotation applied. Am I right?