search

beir-cellar / beir

A Heterogeneous Benchmark for Information Retrieval. Easy to use, evaluate your models across 15+ diverse IR datasets.
http://beir.ai
Apache License 2.0
1.55k stars 186 forks source link

TypeError when running Quick Example #66

Open ReyonRen opened 2 years ago

ReyonRen commented 2 years ago

Thanks for the great work.

When I run the Quick Example, I got a TypeError:

Traceback (most recent call last):
  File "demo.py", line 34, in <module>
    ndcg, _map, recall, precision = retriever.evaluate(qrels, results, retriever.k_values)
  File "/home/renruiyang/anaconda3/envs/beir/lib/python3.6/site-packages/beir/retrieval/evaluation.py", line 63, in evaluate
    evaluator = pytrec_eval.RelevanceEvaluator(qrels, {map_string, ndcg_string, recall_string, precision_string})
TypeError: Unable to resolve all measures.

Could you help me solve this error?

Thanks a lot!

thakur-nandan commented 2 years ago

Hi @ReyonRen, I'm unsure why the error occurs! When I seem to run the custom example, it works fine for me. Here is a Google colab demo: https://colab.research.google.com/drive/1JKySGIMPRAaryVoL3BKkkfO2IRWtkxwB?usp=sharing

Let me know if you still find any issues!

Kind Regards, Nandan Thakur