Open LeMoussel opened 1 week ago
The issue is that this model specifies a prompt, but in MTEB, we have different prompts for tasks, which causes an error. Since this is an instruction model, it would be better to use it with InstructWrapper. Example for e5-instruct models.
OK. I do this;
# https://huggingface.co/jinaai/jina-embeddings-v3/discussions/75
MODEL_NAME = "HIT-TMG/KaLM-embedding-multilingual-mini-instruct-v1"
MODEL_URL = 'https://huggingface.co/HIT-TMG/KaLM-embedding-multilingual-mini-instruct-v1'
OUPUT_FOLDER = "results"
mteb_model = mteb.get_model(
MODEL_NAME,
device="cuda" if torch.cuda.is_available() else "cpu",
)
tasks = mteb.get_tasks(
tasks=TASK_LIST, languages=["fra"]
)
evaluation = mteb.MTEB(tasks=tasks)
mteb_results = evaluation.run(
mteb_model,
eval_splits=["test"],
output_folder=f"{OUPUT_FOLDER}/{MODEL_NAME}",
)
How can I use InstructWrapper
in this case?
You can run this model like this
import mteb
from mteb.models.instruct_wrapper import instruct_wrapper
mteb_model = instruct_wrapper(
model_name_or_path="HIT-TMG/KaLM-embedding-multilingual-mini-instruct-v1",
instruction_template="Instruct: {instruction} \n Query: ",
attn="cccc",
pooling_method="mean",
mode="embedding",
normalized=True,
)
tasks = mteb.get_tasks(
tasks=["SciDocsRR"]
)
evaluation = mteb.MTEB(tasks=tasks)
mteb_results = evaluation.run(
mteb_model,
)
It would be very nice if you could add this model to the models folder with the filled metadata
It is my pleasure to help you by adding this model to the models folder with the metadata filled in, but I am new to using MTEB. What should I do to add this model to the models folder with the metadata filled in?
You should fill in the information similar to the e5_instruct models and run some tasks to ensure that this implementation matches the author's.
With HIT-TMG/KaLM-embedding-multilingual-mini-instruct-v1, I have the following error: