search

embeddings-benchmark / mteb

MTEB: Massive Text Embedding Benchmark
https://arxiv.org/abs/2210.07316
Apache License 2.0
1.85k stars 247 forks source link

New benchmark interface #1272

Open KennethEnevoldsen opened 22 hours ago

KennethEnevoldsen commented 22 hours ago

I have considered if we might want the interface to be:

import mteb

bench = mteb.Benchmark(tasks=[...])
bench.evaluate(model)

bench = mteb.get_benchmark("MTEB(eng)")
bench.evaluate(model)

Since the mteb.MTEB object is kinda of a benchmark object. (not something we should do now)

This would deprecate the mteb.MTEB object.

isaac-chung commented 11 hours ago

I suppose the current mteb.MTEB object is a more flexibly Benchmark. I am in favour of this and v2.0 would be fitting.