-
I have considered if we might want the interface to be:
```
import mteb
bench = mteb.Benchmark(tasks=[...])
bench.evaluate(model)
bench = mteb.get_benchmark("MTEB(eng)")
bench.evaluate(model…
-
Amazing work! It'd be great to have the benchmark integrated in MTEB (https://github.com/embeddings-benchmark/mteb) if you're interested :) I think it'd be great for the community & visibility of your…
-
Hi everyone!
Thanks for developing LLM2Vec and making the source code available.
I was trying to reproduce `LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-supervised` and train a model based on Llama 3.1…
-
When listing all available benchmarks, we wish to
- list the MTEB ones first, sort those by name
- then sort the rest by name
From https://github.com/embeddings-benchmark/mteb/pull/1256#pullreque…
-
Cool work! It'd be great to have it integrated in MTEB (https://github.com/embeddings-benchmark/mteb) if you're interested :)
-
I am using this model `Snowflake/snowflake-arctic-embed-m-v1.5` to reproduce the MTEB leaderboard results on the TRECCOVID dataset. Based on [the leaderboard](https://huggingface.co/spaces/mteb/leader…
-
Really cool work! In case you're interested I think it'd be great for the community & visibility of your project to have InstructIR available in MTEB and maybe also add a leaderboard tab for it 😊 We r…
-
We currently have only one aggregated task (CQGDupstack), however, we can def. imagien more in the future (e.g. for CoIR in https://github.com/embeddings-benchmark/leaderboard/pull/27).
A proposed …
-
Dear MTEB Team,
I noticed that there are some incompatible task names between [github repo](https://github.com/embeddings-benchmark/mteb/tree/main/mteb/tasks) and [leaderboard config](https://githu…
-
Hello, I'm trying to run MTEB on a cluster without internet access, but I am struggling. Here are the following instructions I've followed:
1. `$ pip install mteb`
2. `$ !pip install --upgrade git…