search

QVPR / Patch-NetVLAD

Code for the CVPR2021 paper "Patch-NetVLAD: Multi-Scale Fusion of Locally-Global Descriptors for Place Recognition"
MIT License
517 stars 72 forks source link

how to use pool=patchnetvlad to train the network #27

Closed qzhai closed 2 years ago

qzhai commented 2 years ago

Dear,

Thank you for the great job!

I got a question about how to use the output of patchnetvlad to train the network while vlad_local and vlad_global are generated at the same time. is it OK to use the vlad_global feature to estimate the triplet loss?

Best, Qiang

Tobias-Fischer commented 2 years ago

Hi Qiang,

Many thanks for your interest in Patch-NetVLAD. I am not sure I follow your question. Indeed the vlad_global feature is used for training (see https://github.com/QVPR/Patch-NetVLAD/blob/4edee2f3e4e0173b101ae5b4506151e7cdb899d0/patchnetvlad/training_tools/train_epoch.py#L39). The local features are not being used during training, only at inference.

Feel free to re-open if you have more questions.

Best, Tobi

qzhai commented 2 years ago

Hi Tobi,

Thank you for your reply and sorry for the confusion. I mean Patch-NetVLAD extract both vlad_local and vlad_global (see L196 in patchnetvlad.py) and the released code is used to train NetVLAD.

If I want to train the Patch-NetVLAD, I would modify the Line80 in train_epoch.py as follows:

_, vlad_encoding = model.pool(image_encoding)

is it right?

Best, Qiang

Tobias-Fischer commented 2 years ago

There should not be any changes needed to train Patch-NetVLAD; just run train.py as is. Do you get an error?

As you can see in https://github.com/QVPR/Patch-NetVLAD/blob/main/patchnetvlad/configs/train.ini we use netvlad pooling (not patchnetvlad) when training. Therefore, the code in patchnetvlad.py does not run, but instead the one in netvlad.py. Does that make sense? Again, we do not make use of local features during training. Please make sure that you load the correct config file for training.

qzhai commented 2 years ago

Yes, you are right, I load the default params(pooling=netvlad) from config/train.ini, and it works. what I mean is that can we use the patchnetvlad.py to train the Patch-NetVLAD, not the netvlad.py? that is not a bug, but a hypothetical experiment of mine.

Tobias-Fischer commented 2 years ago

I see what you mean now. Training with patch features in this way won't work out of the box. You would need to make sure that you have ground-truth correspondences between the patches (not just images as is the case currently); and we are not sure how the triplet loss would need to be modified. In other words, training patch-based features is very much still an open research question; it could lead to further significant performance benefits but the implementation would not be straightforward at all.