search

embeddings-benchmark / leaderboard

Code for the MTEB leaderboard
https://hf.co/spaces/mteb/leaderboard
9 stars 6 forks source link

Leaderboard CI failing due to "bright" leaderboard #15

Closed orionw closed 1 month ago

orionw commented 1 month ago

I noticed that I had some emails about the nightly leaderboard failing which are failing to run due to the "bright" leaderboard. I'm not sure what this leaderboard is but it appears there are duplicates in the dataframe which throw an error.

@Muennighoff I see that you added "bright" in commit bb5f6, do you know what might be happening?

Muennighoff commented 1 month ago

Not sure why but it's passing now: https://github.com/embeddings-benchmark/leaderboard/commit/f45efd66996a1dafb9602bd1c6756f393fe8a1ba 🤔

orionw commented 1 month ago

Seems to be a weird issue where it works on commits (I see the recent RAR-B one passed) but the nightly update fails. Recent failure:

if model.library_name == "sentence-transformers" or "sentence-transformers" in model.tags or "modules.json" in {file.rfilename for file in model.siblings}:
TypeError: 'NoneType' object is not iterable

It's exactly the same test, so I wonder if it's something about using the main branch vs using the new branch? Will add it to the list to look at.