Closed matprst closed 2 years ago
Hi @matprst I'm not sure about problems on the HuggingFace part, but for Arguana, it seems that there's a bug with FaissDenseRetrieval, where the line that removes the query id from the result list is not present. This is needed because in Arguana (and quora) queries are a part of the corpus, but not a part of the qrel for itself, which is not an usual setting and can lead to problems like this. The same bug was present on ColBERT evaluation #67.
Thanks for the answer @cadurosar !
Concerning my other problem, I figured out the issue. In encode_corpus()
you are supposed to normalize the embedding yourself when the argument normalize_embeddings
is set to true. I find it a bit counter intuitive since this line suggests the normalization happens "hunder the hood" when using a cosine similarity function. Note that this is only an issue when the search uses a subclass of DenseRetrievalFaissSearch
since the other retrievers do not use the normalize_embeddings
argument when calling encode_corpus
.
Here is my updated encode_corpus
function:
def encode_corpus(self, corpus: List[Dict[str, str]], batch_size: int, **kwargs) -> np.ndarray:
embeddings = []
for i in range(0, len(corpus), batch_size):
# concatenate title and text of the documents in the batch
title_abs = [(doc["title"] + " " + doc["text"]).strip() if "title" in doc else doc["text"].strip() for doc in corpus[i:i+batch_size]]
# preprocess the input
inputs = self.tokenizer(title_abs, padding=True, truncation=True, return_tensors="pt", max_length=512).to(self.device)
result = self.model(**inputs)
if self.pooling:
embedding = self.mean_pooling(result, inputs['attention_mask'])
else: # use [cls] token
embedding = result[0][:, 0, :]
if kwargs.get('normalize_embeddings', False):
embedding = torch.nn.functional.normalize(embedding, p=2, dim=1)
embeddings.append(embedding.cpu().detach().numpy())
return np.vstack(embeddings)
Hi! :smiley: I am trying to reproduce the evaluation of
msmarco-MiniLM-L-6-v3
on BEIR. When I load the model using SentenceBERT I get similar results as the ones on the leaderboard except forArguAna
. However, when I load the same model but using Hugging Face, the results are quite different.Here is my current setup to evaluate the SentenceBERT model (mostly taken from evaluate_faiss_dense.py):
When I load the model from Hugging Face, I need to implement
encode_queries
andencode_corpus
. I also encode the text with mean pooling as shown in the Hugging Face model card:I tested both with mean pooling (as the leaderboard specify) and without. The results without pooling are better than the ones with mean pooling, but still not as good as the ones on the leaderboard. To me, the difference between Hugging Face and SentenceBERT could come from either my implementation of the encoding functions being wrong, or the way I do the pooling.
I have two questions:
ArguAna
? (see first 2 lines of the table above)PS: thanks for the great library, it's awesome!