search

beir-cellar / beir

A Heterogeneous Benchmark for Information Retrieval. Easy to use, evaluate your models across 15+ diverse IR datasets.
http://beir.ai
Apache License 2.0
1.49k stars 177 forks source link

mt5-base model (unicamp-dl/mt5-base-en-msmarco) throws 'nan' outputs #173

Open cramraj8 opened 1 month ago

cramraj8 commented 1 month ago

Hi,

I was running unicamp-dl/mt5-base-en-msmarco: ['▁no' , '▁yes'] model for both English and other My.TyDi languages, but the output scores are nan. When I switched to unicamp-dl/mt5-13b-mmarco-100k: ['▁', '▁true'] model, I get actual real logits. I wonder if there's any issues with underlying unicamp-dl/mt5-base-en-msmarco model.

Thanks.