Open sandeep-krutrim opened 1 month ago
Hi @sandeep-krutrim,
Currently the evaluation only supports the models that we have released on huggingface, supporting custom models will require modifying the mteb package, specifically llm2vec_models.py file.
Hi @sandeep-krutrim,
is there any progress in evaluating on MTEB for locally trained LLM2Vec models? Specifically, I would like to train a version of Llama 3.1 and get evaluation results on MTEB.
In case there is not, @vaibhavad could you maybe provide some more specific pointers to what would need to be changed in the llm2vec_models.py file to evaluate models trained according to your Readme? Or is there any other way that allows a way to evaluate custom trained models?
Thank you!
I have trained a model using supervised contrastive. I saved the model using -
l2v.save('/llm2vec_models/final_merged_model', merge_before_save=True, save_config=True)
Now when I try to run mteb_eval.py using -
!python experiments/mteb_eval.py --model_name model_name \ --task_name STS16 \ --task_to_instructions_fp test_configs/mteb/task_to_instructions.json \ --output_dir results
I get the error -
`WARNING:sentence_transformers.SentenceTransformer:No sentence-transformers model found with name sentence-transformers/model_name. Creating a new one with mean pooling. Traceback (most recent call last): File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/huggingface_hub/utils/_errors.py", line 304, in hf_raise_for_status response.raise_for_status() File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/requests/models.py", line 1024, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://huggingface.co/sentence-transformers/model_name/resolve/main/config.json
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/transformers/utils/hub.py", line 402, in cached_file resolved_file = hf_hub_download( File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn return fn(*args, *kwargs) File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 1221, in hf_hub_download return _hf_hub_download_to_cache_dir( File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 1325, in _hf_hub_download_to_cache_dir _raise_on_head_call_error(head_call_error, force_download, local_files_only) File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 1823, in _raise_on_head_call_error raise head_call_error File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 1722, in _get_metadata_or_catch_error metadata = get_hf_file_metadata(url=url, proxies=proxies, timeout=etag_timeout, headers=headers) File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn return fn(args, **kwargs) File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 1645, in get_hf_file_metadata r = _request_wrapper( File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 372, in _request_wrapper response = _request_wrapper( File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 396, in _request_wrapper hf_raise_for_status(response) File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/huggingface_hub/utils/_errors.py", line 352, in hf_raise_for_status raise RepositoryNotFoundError(message, response) from e huggingface_hub.utils._errors.RepositoryNotFoundError: 401 Client Error. (Request ID: Root=1-6690ade6-4a6e9ced7aa95d064f357ee5;f3ba928a-c379-4be1-8ae2-6748cf28a965)
Repository Not Found for url: https://huggingface.co/sentence-transformers/model_name/resolve/main/config.json. Please make sure you specified the correct
repo_id
andrepo_type
. If you are trying to access a private or gated repo, make sure you are authenticated. Invalid username or password.The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/home/sandeep.pandey/august15/llm2vec/experiments/mteb_eval.py", line 28, in
model = mteb.get_model(args.model_name, model_kwargs)
File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/mteb/models/init.py", line 42, in get_model
meta = get_model_meta(model_name, revision)
File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/mteb/models/init.py", line 77, in get_model_meta
model = SentenceTransformer(
File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/sentence_transformers/SentenceTransformer.py", line 299, in init
modules = self._load_auto_model(
File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/sentence_transformers/SentenceTransformer.py", line 1324, in _load_auto_model
transformer_model = Transformer(
File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/sentence_transformers/models/Transformer.py", line 53, in init
config = AutoConfig.from_pretrained(model_name_or_path, config_args, cache_dir=cache_dir)
File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/transformers/models/auto/configuration_auto.py", line 965, in from_pretrained
config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, kwargs)
File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/transformers/configuration_utils.py", line 632, in get_config_dict
config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, kwargs)
File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/transformers/configuration_utils.py", line 689, in _get_config_dict
resolved_config_file = cached_file(
File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/transformers/utils/hub.py", line 425, in cached_file
raise EnvironmentError(
OSError: sentence-transformers/model_name is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo either by logging in with
huggingface-cli login
or by passingtoken=<your_token>