castorini / pygaggle

a gaggle of deep neural architectures for text ranking and question answering, designed for Pyserini
http://pygaggle.ai/
Apache License 2.0
329 stars 97 forks source link

[Bug?] uploaded monot5-large-msmarco-10k and monot5-large-msmarco model is exactly the same #330

Open soyoung97 opened 8 months ago

soyoung97 commented 8 months ago

Hello, I was trying to evaluate the monot5-large model on the beir benchmark, but I noticed that the uploaded models - 10k and 100k models - has exactly the same model. (when I do diff on terminal, it shows no difference) image

Also, the performance of large models is a lot lower than the performance of base models - Could you check if the two uploaded large models are accurate? (maybe relevant to the previous issue)

Thank you!

soyoung97 commented 8 months ago

Another example showing the weight of two models is exactly the same: (code from here)

image
KID-22 commented 6 months ago

I ran into the same problem, I tested the 10K and 100K monoT5 on BEIR and found the results were the same, and both were very low, close to zero