Open jordane95 opened 2 years ago
As hostname I think you must use http://localhost (or http://localhost:9200), not just localhost
Thank you so much! I change the hostname to http://localhost:9200
and it works. But when I run it to evaluate BM25, I get different scores at different runs. For example, the NDCG@10 score ranges from 0.64~0.67 on scifact dataset. Do you know why? Is there any randomness in the BM25 algorithm?
This was addressed in https://github.com/UKPLab/beir/issues/58
Not sure if the latest release already includes this. You can either update BEIR to use the latest version from the GIT. Or you add a sleep after you index the documents in your code.
I see. It's fixed in the beir
code but not yet included in the examples
. I add a sleep time and eventually get a consistent score.
Hi @jordane95,
Yes soon with our next pip update, hopefully, this should not be an issue anymore and consistent scores should be visible with Elasticsearch BM25. Thanks for notifying me!
Kind regards, Nandan Thakur
Hi, I was trying to run your
evaluate_bm25.py
baseline, but I got the following error. There may be some problem withelasticsearch
. Could you please help me fix it?