staoxiao
commented
2 months ago Qwen2Model is not an embedding model, so it cannot be used to evaluate C-METB.
Open ChiZhang-bit opened 2 months ago
staoxiao
commented
2 months ago Qwen2Model is not an embedding model, so it cannot be used to evaluate C-METB.
在模型调用的时候，会出现没有encode方法的报错。使用的是QWen0.5B模型：
Error while evaluating CmedqaRetrieval: 'Qwen2Model' object has no attribute 'encode' Traceback (most recent call last): File "/data2/mteb/zhangchi/mteb_eval/run_mteb_chinese.py", line 266, in <module> evaluation.run(model, output_folder=f"qwen/qwen_MNR_supervised_v1") File "/home/zhouzhou/anaconda3/envs/llm2vec/lib/python3.9/site-packages/mteb/evaluation/MTEB.py", line 271, in run raise e File "/home/zhouzhou/anaconda3/envs/llm2vec/lib/python3.9/site-packages/mteb/evaluation/MTEB.py", line 253, in run results = task.evaluate(model, split, **kwargs) File "/home/zhouzhou/anaconda3/envs/llm2vec/lib/python3.9/site-packages/mteb/abstasks/AbsTaskRetrieval.py", line 101, in evaluate results = retriever.retrieve(corpus, queries) File "/home/zhouzhou/anaconda3/envs/llm2vec/lib/python3.9/site-packages/beir/retrieval/evaluation.py", line 20, in retrieve return self.retriever.search(corpus, queries, self.top_k, self.score_function, **kwargs) File "/home/zhouzhou/anaconda3/envs/llm2vec/lib/python3.9/site-packages/beir/retrieval/search/dense/exact_search.py", line 42, in search query_embeddings = self.model.encode_queries( File "/home/zhouzhou/anaconda3/envs/llm2vec/lib/python3.9/site-packages/mteb/abstasks/AbsTaskRetrieval.py", line 154, in encode_queries return self.model.encode(queries, batch_size=batch_size, **kwargs) File "/home/zhouzhou/anaconda3/envs/llm2vec/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1688, in __getattr__ raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'") AttributeError: 'Qwen2Model' object has no attribute 'encode'