search

embeddings-benchmark / mteb

MTEB: Massive Text Embedding Benchmark
https://arxiv.org/abs/2210.07316
Apache License 2.0
1.93k stars 264 forks source link

RetrievalEvaluator NaN values for similarity scores #1389

Open gnatesan opened 1 day ago

gnatesan commented 1 day ago

I am looking at RetrievalEvaluator.py and I see the line cos_scores[torch.isnan(cos_scores)] = -1

Basically, if a query-document similarity score has a NaN value, then it gets changed to -1.

My question is how would a NaN score appear when using cosine similarity as the distance metric? Would the query and document embedding vectors contain very small values when a NaN value appears?

KennethEnevoldsen commented 1 day ago

My guess is that it can appear as a result of an invalid model output. @Muennighoff or @orionw might know more about it? We should probably log a warning around that statement.

gnatesan commented 18 hours ago

When you say result of an invalid model output, are you referring to the output similarity score function or the generation of text embeddings?

KennethEnevoldsen commented 10 hours ago

Generation of the text embedding. Adding logging to in #1398. Ran through tests, but couldn't find a case where it was actually used.