search

McGill-NLP / llm2vec

Code for 'LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders'
https://mcgill-nlp.github.io/llm2vec/
MIT License
1.07k stars 78 forks source link

Unable to load merged model for MTEB evaluation #123

Open sandeep-krutrim opened 1 month ago

sandeep-krutrim commented 1 month ago

I have trained a model using supervised contrastive. I saved the model using -

l2v.save('/llm2vec_models/final_merged_model', merge_before_save=True, save_config=True)

Now when I try to run mteb_eval.py using - !python experiments/mteb_eval.py --model_name model_name \ --task_name STS16 \ --task_to_instructions_fp test_configs/mteb/task_to_instructions.json \ --output_dir results

I get the error -

`WARNING:sentence_transformers.SentenceTransformer:No sentence-transformers model found with name sentence-transformers/model_name. Creating a new one with mean pooling. Traceback (most recent call last): File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/huggingface_hub/utils/_errors.py", line 304, in hf_raise_for_status response.raise_for_status() File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/requests/models.py", line 1024, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://huggingface.co/sentence-transformers/model_name/resolve/main/config.json

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/transformers/utils/hub.py", line 402, in cached_file resolved_file = hf_hub_download( File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn return fn(*args, *kwargs) File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 1221, in hf_hub_download return _hf_hub_download_to_cache_dir( File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 1325, in _hf_hub_download_to_cache_dir _raise_on_head_call_error(head_call_error, force_download, local_files_only) File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 1823, in _raise_on_head_call_error raise head_call_error File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 1722, in _get_metadata_or_catch_error metadata = get_hf_file_metadata(url=url, proxies=proxies, timeout=etag_timeout, headers=headers) File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn return fn(args, **kwargs) File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 1645, in get_hf_file_metadata r = _request_wrapper( File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 372, in _request_wrapper response = _request_wrapper( File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/huggingface_hub/file_download.py", line 396, in _request_wrapper hf_raise_for_status(response) File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/huggingface_hub/utils/_errors.py", line 352, in hf_raise_for_status raise RepositoryNotFoundError(message, response) from e huggingface_hub.utils._errors.RepositoryNotFoundError: 401 Client Error. (Request ID: Root=1-6690ade6-4a6e9ced7aa95d064f357ee5;f3ba928a-c379-4be1-8ae2-6748cf28a965)

Repository Not Found for url: https://huggingface.co/sentence-transformers/model_name/resolve/main/config.json. Please make sure you specified the correct repo_id and repo_type. If you are trying to access a private or gated repo, make sure you are authenticated. Invalid username or password.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/sandeep.pandey/august15/llm2vec/experiments/mteb_eval.py", line 28, in model = mteb.get_model(args.model_name, model_kwargs) File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/mteb/models/init.py", line 42, in get_model meta = get_model_meta(model_name, revision) File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/mteb/models/init.py", line 77, in get_model_meta model = SentenceTransformer( File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/sentence_transformers/SentenceTransformer.py", line 299, in init modules = self._load_auto_model( File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/sentence_transformers/SentenceTransformer.py", line 1324, in _load_auto_model transformer_model = Transformer( File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/sentence_transformers/models/Transformer.py", line 53, in init config = AutoConfig.from_pretrained(model_name_or_path, config_args, cache_dir=cache_dir) File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/transformers/models/auto/configuration_auto.py", line 965, in from_pretrained config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, kwargs) File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/transformers/configuration_utils.py", line 632, in get_config_dict config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, kwargs) File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/transformers/configuration_utils.py", line 689, in _get_config_dict resolved_config_file = cached_file( File "/home/sandeep.pandey/miniconda3/envs/llm2vec_env/lib/python3.9/site-packages/transformers/utils/hub.py", line 425, in cached_file raise EnvironmentError( OSError: sentence-transformers/model_name is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models' If this is a private repository, make sure to pass a token having permission to this repo either by logging in with huggingface-cli login or by passing token=<your_token>

vaibhavad commented 1 month ago

Hi @sandeep-krutrim,

Currently the evaluation only supports the models that we have released on huggingface, supporting custom models will require modifying the mteb package, specifically llm2vec_models.py file.

stefanhgm commented 2 weeks ago

Hi @sandeep-krutrim,

is there any progress in evaluating on MTEB for locally trained LLM2Vec models? Specifically, I would like to train a version of Llama 3.1 and get evaluation results on MTEB.

In case there is not, @vaibhavad could you maybe provide some more specific pointers to what would need to be changed in the llm2vec_models.py file to evaluate models trained according to your Readme? Or is there any other way that allows a way to evaluate custom trained models?

Thank you!