search

embeddings-benchmark / leaderboard

Code for the MTEB leaderboard
https://hf.co/spaces/mteb/leaderboard
15 stars 10 forks source link

[Bugs] Incompatible task names between github repo and leaderboard config #23

Open hanhainebula opened 3 months ago

hanhainebula commented 3 months ago

Dear MTEB Team,

I noticed that there are some incompatible task names between github repo and leaderboard config:

The above bugs will cause the corresponding results in the model card generated with the latest mteb package can not be loaded into the leaderboard. Therefore, our latest model bge-multilingual-gemma2 now has some empty results in the leaderboard. I just modified the model card to adapt to the leaderboard config. Could you update the leaderboard again to load the missing results? Thank you very much!

hanhainebula commented 3 months ago

Thank you for the update! But I'm so sorry for the correction:

There's still one missing result in MTEB-fr. Could you update the leaderboard again? Thank you!

hanhainebula commented 3 months ago

The metric used for BSARDRetrieval is also incompatiable: