search

embeddings-benchmark / leaderboard

Code for the MTEB leaderboard
https://hf.co/spaces/mteb/leaderboard
5 stars 2 forks source link

Automatic update tasks list #4

Open Samoed opened 2 weeks ago

Samoed commented 2 weeks ago

How about automating the update of the tasks list from the mteb library? A separate script could be created to run before refresh.py to handle this task.

@orionw

orionw commented 2 weeks ago

Hi @Samoed, would love to make it more automatic. For most of the leaderboard though the tasks are pre-defined from some set of results (e.g. English MTEB is pre-defined, French is pre-defined, etc.). I think the results objects that gets pulled in has all the tasks that are run, but then we filter them in the refresh.py script to the benchmarks we care about.

We do want to transition to a "create-your-own" style leaderboard where you can select the tasks though. And for that automatically updating the tasks seems very important!

Perhaps I'm misunderstanding - let me know if this is not what you were meaning?

Samoed commented 2 weeks ago

My intention was to automatically include tasks for each language from mteb. However, if you're currently using only a subset of those tasks, then my suggestion may not be applicable for this repository, but for suitable for create-your-own.