Improve leaderboard 2.0 readability

[KennethEnevoldsen]

A couple of comments for readability:

• [x] remove org from model name
• [x] and add reference link #1316
• [x] multiply scores by 100 and keep one decimal, e.g. 78.1 (@orionw not sure if this also works for followIR?). I believe anything more precise does not translate to a meaningful performance difference
• [x] It might be ideal to move the "performance pr. task" to a separate table in a fold-down menu.
• [x] We could also add highlighting of highest scores using styling elements
• [x] Add ranks to summary table
• [x] Replace average rank with borda count and rank (#1344 )

Originally posted by @KennethEnevoldsen in https://github.com/embeddings-benchmark/mteb/issues/1312#issuecomment-2435013987

[x-tabdeveloping]

By fold-down menu you mean an accordion right?

[KennethEnevoldsen]

It was my initial idea yes, but I suppose multiple things could work - tabs would also be an option:

[x-tabdeveloping]

That ain't dumb! might try that one then.

[orionw]

multiply scores by 100 and keep one decimal, e.g. 78.1 (@orionw not sure if this also works for followIR?)

It does work for FollowIR!

Also is the v2 leaderboard up somewhere or is this a picture from development?

[x-tabdeveloping]

It's still in development. I'm using the leaderboard_2, brnach for new changes. You can run it by:

from mteb.leaderboard import demo

demo.launch()

[x-tabdeveloping]

I can host a demo version on my HF profile btw if it's something we'd be interested in having @orionw

[orionw]

Ah, no problem @x-tabdeveloping! For some reason I misunderstood and thought it was already up. Thanks for the offer, but no need to add extra work during your development. It’s looking great already though! 🚀

[x-tabdeveloping]

Here's a demo of the current version: https://huggingface.co/spaces/kardosdrur/mmteb_leaderboard_demo

[tomaarsen]

Thanks for sharing the dev version!

[Muennighoff]

The leaderboard looks really amazing! Probably already planned but

• some indication of contamination would be great as we discussed (maybe we just manually add it to the metadata for now where we know it (e.g. trained_on_{task_name}_{task_split}: true or training_datasets: [(Emotion, train), (Amazon, test), ...] or something else) and invite users to update the metadata via PR)
• maybe adding some statistics, e.g. on the current LB we have the below at the bottom

  Total Datasets: 213
  Total Languages: 113
  Total Scores: 88857
  Total Models: 469

  (could be auto-displayed per-benchmark when selecting a benchmark)

• Maybe we want to link it with the arena somehow? (e.g. one dropdown option could be the arena and it links to the arena space; or we just have a banner at the bottom or top to motivate people to checkout the arena or similar; you probably have better ideas!)

[x-tabdeveloping]

@Muennighoff I'm on it!

[x-tabdeveloping]

Hey @Muennighoff what does Total scores mean?

[Muennighoff]

Total scores is the total number of scores i.e. how many numbers there are in the table. Maybe there's a better name for it 🤔

[KennethEnevoldsen]

Might be worth moving integration with Arena to a separate issue (It might work well with #1432). I think it might warrant some more discussion. To begin with we could also add it to the description of MTEB(eng, beta). Something like:

"English also has an arena-style benchmark for evaluating embeddings. You can check this out here".

[x-tabdeveloping]

I'm a bit stopped in my tracks because of glaring issues with Gradio's dataframes (1, 2). I have implemented the plot though, and will add overview info to the benchmarks' descriptions.