Open AbhishekBose opened 3 weeks ago
Hello!
I was under the impression that sentence-transformers uses rust's bindings
Sentence Transformers relies on torch
instead to run its models. That would cause the discrepancy.
Having said that, on GPUs the candle-based runtime should be quicker, see https://github.com/huggingface/text-embeddings-inference which uses Candle. Perhaps that project has quicker runtimes because of other optimizations (like Flash Attention), or perhaps because it tests on larger models like https://huggingface.co/BAAI/bge-base-en-v1.5?
Flash attention is a possibility. Gotta check that part Although I did the test using the same model, which was 'sentence-transformers/all-MiniLM-L6-v2'
Here is my candle implementation: (Taken from the examples itself)
`pub fn encode(&self, prompt: &str) -> Result<(Tensor,Tensor)> {
and here is the python implementation
model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2') embeddings = model.encode([request.query])
On comparing just the encoding times, this is what I get
Encoding time for rust:
Encoding time for python:
What could be the reason for the difference in inference time between the two? I was under the impression that sentence-transformers uses rust's bindings
The experiment was run on an m1 mac air