search

FlagOpen / FlagEmbedding

Retrieval and Retrieval-augmented LLMs
MIT License
7.71k stars 561 forks source link

加载本地的数据，对嵌入模型进行测评，报错 #1243

Open chaoStart opened 1 week ago

chaoStart commented 1 week ago

企业微信截图_17319963789656

Traceback (most recent call last): File "/home/bqc/project/evalscope/MTEBtest.py", line 30, in evaluation.run(model, output_folder="BAAI",verbosity=2,eval_splits=["test"]) File "/home/bqc/.conda/envs/evalscope/lib/python3.10/site-packages/mteb/evaluation/MTEB.py", line 477, in run raise e File "/home/bqc/.conda/envs/evalscope/lib/python3.10/site-packages/mteb/evaluation/MTEB.py", line 407, in run task.load_data(eval_splits=task_eval_splits, **kwargs) File "/home/bqc/.conda/envs/evalscope/lib/python3.10/site-packages/mteb/tasks/Retrieval/zho/CMTEBRetrieval.py", line 72, in load_data self.corpus, self.queries, self.relevant_docs = load_retrieval_data( File "/home/bqc/.conda/envs/evalscope/lib/python3.10/site-packages/mteb/tasks/Retrieval/zho/CMTEBRetrieval.py", line 15, in load_retrieval_data qrels = load_dataset(dataset_path + "-qrels", revision=qrel_revision)[eval_split] File "/home/bqc/.conda/envs/evalscope/lib/python3.10/site-packages/datasets/load.py", line 2074, in load_dataset builder_instance = load_dataset_builder( File "/home/bqc/.conda/envs/evalscope/lib/python3.10/site-packages/datasets/load.py", line 1795, in load_dataset_builder dataset_module = dataset_module_factory( File "/home/bqc/.conda/envs/evalscope/lib/python3.10/site-packages/datasets/load.py", line 1677, in dataset_module_factory raise FileNotFoundError(f"Couldn't find any data file at {relative_to_absolute_path(path)}.") FileNotFoundError: Couldn't find any data file at /home/bqc/.cache/evalscope/dataset/T2Retrieval-qrels.

chaoStart commented 1 week ago

下面是本人的测评代码

from sentence_transformers import SentenceTransformer model_name = "BAAI/bge-base-en-v1.5" model = SentenceTransformer(model_name_or_path="/home/bqc/xinference_LLM/bge-large-zh-v1.5",local_files_only=True)

if name == 'main': import mteb from mteb import MTEB ChineseTaskList = ["T2Retrieval"] tasks = mteb.get_tasks(tasks=ChineseTaskList)

for task in tasks:
    evaluation = MTEB(tasks=[task])
    evaluation.run(model,output_folder=f"zh_results/{model_name}",verbosity=2,eval_splits=["test"])
hanhainebula commented 4 days ago

你好，@chaoStart。可以参考我们在这里提供的示例进行评测：https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/evaluation#1-MTEB

如果因为网络问题无法访问 huggingface.co，可以设置环境变量 export HF_ENDPOINT="hf-mirror.com" 来从镜像加载数据。