Open akjindal53244 opened 5 months ago
Hello!
If I remember correctly, https://huggingface.co/intfloat/e5-mistral-7b-instruct is not compatible with Sentence Transformers out of the box due to its left-sided tokenizer. The Sentence Transformers Pooler isn't expecting such a tokenizer, so it incorrectly picks the "last" token. With other words, it's best to stick to the usage in the README until there's left-sided tokenizer support in Sentence Transformers. Apologies for the inconvenience.
@akjindal53244 You need to pass add_eos_token=True when creating the sentence-transformers model
word_embedding_model = Transformer(model_name_or_path="intfloat/e5-mistral-7b-instruct", tokenizer_args={"add_eos_token": True})
last_token_pooling_model = Pooling(4096, pooling_mode="lasttoken")
model = SentenceTransformer(modules=[word_embedding_model, last_token_pooling_model], device="cpu")
embeddings = model.encode(input_texts, normalize_embeddings=True, device="cpu")
similarities = (embeddings[:2] @ embeddings[2:].T)
print(similarities)
Output:
[[0.82913667 0.4797157 ]
[0.4695111 0.81743485]]
Oh, wow! Great job. Do you think we should make a pull request on https://huggingface.co/intfloat/e5-mistral-7b-instruct to add "add_eos_token": true
in the tokenizer_config.json
?
Then the transformers-based usage snippet can be simplified and we can integrate Sentence Transformers out of the box!
cc @intfloat as this might interest you
Hi @tomaarsen @satyamk7054 , coincidentally, I was also able to get it working after few hours of opening the issue.
I was able to trace the issue by comparing input_ids
being passed to the model in both codebases and found missing eos
token when using SentenceTransformers and added that through tokenizer_args
argument. Next, added lasttoken
as pooling mechanism. Both these fixed combined reproduced the expected output!
Here is my code:
def load_e5_model(max_seq_length):
# Critical: E5-Mistral adds EOS token explicitly.
tokenizer_args = {
"add_eos_token": True
}
emb_dim = 4096
word_embedding_model = models.Transformer(model_name_or_path=emb_model_name, tokenizer_args = tokenizer_args, max_seq_length=max_seq_length)
# Important: E5-Mistral uses last token embedding
last_token_pooling_model = models.Pooling(emb_dim, pooling_mode = "lasttoken")
model = SentenceTransformer(modules=[word_embedding_model, last_token_pooling_model])
return model
To load the model: model = load_e5_model(max_seq_length)
We should add these changes to the repo as the model is widely used :)
I am using
intfloat/e5-mistral-7b-instruct
model to get last hidden state for my input and compute cosine similarity.I am using a toy example provided at: https://huggingface.co/intfloat/e5-mistral-7b-instruct#usage
Method-1
Code ref: https://huggingface.co/intfloat/e5-mistral-7b-instruct#usage
Output:
[[0.8291366100311279, 0.4797157943248749], [0.46951112151145935, 0.8174349665641785]]
Method-2
Here is my code using
SentenceTransformers
library:Output:
[[0.5713191032409668, 0.20948277413845062], [0.3217913508415222, 0.5535271167755127]]
Output
of both should match but looks like I am missing something obvious. Any help is appreciated!