search

facebookresearch / Qinco

Residual Quantization with Implicit Neural Codebooks
https://arxiv.org/abs/2401.14732
Other
49 stars 2 forks source link

Question regarding to the training objective (loss function). #4

Open EryiXie opened 3 months ago

EryiXie commented 3 months ago

Hi, thank you for this excellent work, I am already using it in my project which shows impressive results.

I just want to ask about the training objective:

https://github.com/facebookresearch/Qinco/blob/0ddfc77f460c1e00ab3d6abbeb3e04c45cb18e23/utils.py#L28-L52

, where l2 distance is used in both functions. I am curious about the choice here: l2 distance vs. cosine similarity. Are there some insights behind using l2 distance here?

Best regards.

mdouze commented 3 months ago

First, cosine similarity and L2 distance are equivalent for normalized vectors (see https://github.com/facebookresearch/faiss/wiki/MetricType-and-distances#how-can-i-index-vectors-for-cosine-similarity), so if queries and database vectors are both normalized, then QINCo will just work as well for cosine as for L2 search.

Regarding this function, although the input vectors are normalized, the centroids are not. Thus, replacing this part of the loss with cosine would yield different results, it's hard to tell what the impact would be.

Intuitively, the centroids should not be normalized because their magnitude decreases with each QINCo step.