-
mteb takes 30s while transformers takes 6s (https://colab.research.google.com/drive/166j9HRQF8hAqwiiYpMDvYoznm1v-veOC?usp=sharing). Probably because of the many `*` imports 🤔
-
Traceback (most recent call last):
File "/home/bqc/project/evalscope/MTEBtest.py", lin…
-
Hi, thanks for the cool MTEB toolkit.
We are currently preparing to release an embedding model for *universal multimodal retrieval*, along with our compiled evaluations. I noticed that you are also…
-
I am trying to reproduce the performance of the model "jina_v3" https://huggingface.co/jinaai/jina-embeddings-v3 on text classificiaton benchmark.
And I am using the code below:
import mteb
from…
-
With [HIT-TMG/KaLM-embedding-multilingual-mini-instruct-v1](https://huggingface.co/HIT-TMG/KaLM-embedding-multilingual-mini-instruct-v1), I have the following error:
```
Loader not specified for m…
-
I am trying to evaluate GritLM-7B on MTEB datasets using the provided script.
```
#!/bin/bash
python /home/e/e1347696/unified_encoder_decoder/src/eval/MTEB/eval_mteb.py \
--model_name_or_pat…
-
Running the tasks with `BAAI/bge-visualized-base-base/m3` and getting errors like below
```
ERROR:mteb.evaluation.MTEB:Error while evaluating InfoSeekIT2TRetrieval: The size of tensor a (516) must m…
-
Hi everyone!
Thanks for developing LLM2Vec and making the source code available.
I was trying to reproduce `LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-supervised` and train a model based on Llama 3.1…
-
```
2024-11-09 21:39:44.994636: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already b…
-
```
ERROR:mteb.evaluation.MTEB:Error while evaluating Caltech101: No module named 'datasets.tasks'
Traceback (most recent call last):
…