-
```
2024-11-09 21:39:44.994636: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already b…
-
Seems like filtering for Cross-Encoders is not working anymore 🤔 Maybe cc @orionw has an idea?
(originally raised here https://huggingface.co/spaces/mteb/leaderboard/discussions/145 cc @guillaumegros…
-
Looks like the import is wrong, and should just be `from utils ...`, importing from the local `utils.py`?
```
Traceback (most recent call last):
File "/workspace/arctic-embed/src/mteb_retrieval…
-
Some time ago I built this benchmark for MMTEB called CodeEditSearchRetrieval: https://github.com/embeddings-benchmark/mteb/blob/93a4db11371f1582c26277f12da84c1107296cd0/mteb/tasks/Retrieval/code/Code…
-
This issue is an overview of tasks to add for a massive multimodal extension of MTEB. The modalities are:
- T=Text
- I=Image
- A=Audio
- V=Video without audio i.e. just multiple images
Below is…
-
- [ ] #1372
- [x] https://github.com/embeddings-benchmark/mteb/issues/1396
- [x] Ensure old results are present in the new leaderboard (https://github.com/embeddings-benchmark/results/pull/43)
- […
-
Cool work! It'd be great to have it integrated in MTEB (https://github.com/embeddings-benchmark/mteb) if you're interested :)
-
Really cool work! In case you're interested I think it'd be great for the community & visibility of your project to have InstructIR available in MTEB and maybe also add a leaderboard tab for it 😊 We r…
-
### Question
I've tried multiple models from MTEB dashboard (e.g. `jinaai/jina-embeddings-v3`, `jinaai/jina-embeddings-v2`, `dunzhang/stella_en_400M_v5`), but none of them work.
It's not clear whi…
-
I have considered if we might want the interface to be:
```
import mteb
bench = mteb.Benchmark(tasks=[...])
bench.evaluate(model)
bench = mteb.get_benchmark("MTEB(eng)")
bench.evaluate(model…