Closed orionw closed 1 month ago
Not sure why but it's passing now: https://github.com/embeddings-benchmark/leaderboard/commit/f45efd66996a1dafb9602bd1c6756f393fe8a1ba 🤔
Seems to be a weird issue where it works on commits (I see the recent RAR-B one passed) but the nightly update fails. Recent failure:
if model.library_name == "sentence-transformers" or "sentence-transformers" in model.tags or "modules.json" in {file.rfilename for file in model.siblings}:
TypeError: 'NoneType' object is not iterable
It's exactly the same test, so I wonder if it's something about using the main branch vs using the new branch? Will add it to the list to look at.
I noticed that I had some emails about the nightly leaderboard failing which are failing to run due to the "bright" leaderboard. I'm not sure what this leaderboard is but it appears there are duplicates in the dataframe which throw an error.
@Muennighoff I see that you added "bright" in commit bb5f6, do you know what might be happening?