ga1i13o
commented
1 year ago hello,
-
traditionally in VPR, during training, images within 10 m are considered suitable to be mined as positives. Whereas during inference, usually matches are considered correct if within 25 mt. The reason during training only images within 10 m are considered is that they have a higher chance of having high visual overlap with the query. However, even if we choose positives within 10 m, we want to avoid using as negatives images in the range (10, 25) meters, as this would push them away from the query; and even if they might have not-so-high visual overlap, they would still be considered correct matches if ever retrieved. For this reason during training we keep this distinction; and soft positives are images within 25 m. We need this distinction so that when we select positives, we take them from hard_positives. When we select negatives, we make sure to not select any of them from the soft_positives list. during inference we use only hard_positives, but changing the threshold to 25 m, as we do not need to do mining .
-
note that this discussion only makes sense for training; for inference we do not care about visual overlap and we are just happy if for whatever reason we retrieva images within 25 m. In VPR there is no clear definition of positive images based on coordinates.Seminal works like netvlad were developed without heading information, and thus relied on mining to select positives: among images within 10 m, we select the one with highest feature similarity to the query. even though more recent datasets like MSLS and SF-XL do provide the heading information, and in the Cosplace paper it is used to build classes, and it does classification rather than metric learning via triplet loss. For retrieval it has never been studied the usage of heading to define positives; for this reason following common practices in literature, we do not try to use heading, simply use 10 m as threshold (during training) for hard positives, and use the network to extract features and pick the closest candidates in feature space, which thus are most likely to have high visual overlap
-
it can happen (and it does in MSLS) that a query sequence does not correspond to any database sequences within 25 m.
-
the parameter 'cut_last_frame' is needed because the way the code that builds sequence is written, it only works with odd sequence lengths (3, 5, 7..) ; thus if you wanted to use even sequences, it can be done by setting cut_last_frame true which then cuts the last frame. although usually 5 is the length used
-
that comment is slightly imprecise. it means that if it is training time, we need to return triplets of query,pos,neg. Instead if it is inference, or we need to initialize netvlad, we will return a single sequence ( 'single image' in the comment)
-
if you look at the code, after the line 328 that you mentioned, that function computes the mining. In the end, it will fill the global_triplets_index.
https://github.com/vandal-vpr/vg-transformers/blob/3947df2469d54aca7dfe3b6f6b5b22c242c1c41b/tvg/datasets/dataset.py#L351
so in the end it contains the indexes of query, positive, and negative sequences
- the 10 m constraint is respected simply because we sample potential positives from the list of hard positives. https://github.com/vandal-vpr/vg-transformers/blob/3947df2469d54aca7dfe3b6f6b5b22c242c1c41b/tvg/datasets/dataset.py#L334
the structure self.pIdx contains for each query sequences the indexes of hard positive sequences
hope I was clear
divyagupta25
Hi, I want to understand the training of CCT384 + SeqVLAD model on MSLS. I have some doubts in dataset loading.
self.triplets_global_indexesis empty? https://github.com/vandal-vpr/vg-transformers/blob/3947df2469d54aca7dfe3b6f6b5b22c242c1c41b/tvg/datasets/dataset.py#L328Could you please help me with these? Thanks in advance!