bge-reranker-base在SciDocRR上MAP很低，正常吗？

[jfzhang726]

运行 import mteb from sentence_transformers import SentenceTransformer

model_name = "BAAI/bge-reranker-base"

model = SentenceTransformer(model_name) tasks = mteb.get_tasks(tasks=["SciDocsRR"]) evaluation = mteb.MTEB(tasks=tasks) results = evaluation.run(model, output_folder=f"results/{model_name}") 得到结果如下： { "dataset_revision": "d3c5e1fc0b855ab6097bf1cda04dd73947d7caab", "evaluation_time": 146.51545810699463, "kg_co2_emissions": null, "mteb_version": "1.12.79", "scores": { "test": [ { "hf_subset": "default", "languages": [ "eng-Latn" ], "main_score": 0.26548822156223856, "map": 0.26548822156223856, "mrr": 0.3757299073475544, "nAUC_map_diff1": 0.024088915166030296, "nAUC_map_max": -0.02239489292976422, "nAUC_map_std": 0.05861038154610513, "nAUC_mrr_diff1": 0.030523666162389315, "nAUC_mrr_max": -0.013564049963031306, "nAUC_mrr_std": 0.039030020508059156 } ] }, "task_name": "SciDocsRR" } 这个结果正常不正常？

[staoxiao]

bge-reranker-base是cross-encoder模型，不能这样使用，请使用FlagReranker。 另外， mteb目前应当是不支持cross-encoder的评估，评估方式是错误的。