vaibhavad
commented
6 months ago Hi @Georgepitt, thanks for your interest in our work.
We also faced this issue during our model development, PEFT library pushed a fix 2 months ago so the latest version should support offline loading.
Here is what I have tried on my end
- First, download the models with internet connection
from huggingface_hub import snapshot_download snapshot_download(repo_id="McGill-NLP/LLM2Vec-Mistral-7B-Instruct-v2-mntp") # this command will return local address, lets call it <MNTP_LOCAL_PATH>
snapshot_download(repo_id="McGill-NLP/LLM2Vec-Mistral-7B-Instruct-v2-mntp-unsup-simcse")
this command will return local address, lets call it
2. In a separate session without the internet, start python interactive/script with `HF_HUB_OFFLINE=1` option
```bash
HF_HUB_OFFLINE=1 python- In this offline python session, this script should work
import torch from llm2vec import LLM2Vec
l2v = LLM2Vec.from_pretrained(
"
Here are details of relevant library versions in my environment
```bash
huggingface-hub 0.22.2
peft 0.10.0
transformers 4.40.1Let me know if you have any further questions.
SouLeo
Georgepitt
GeraldWu23
hch1231
saikot-paul
Hello, the computing cluster provided by the lab needs to run offline. But the code in usage needs to be networked. I have changed the code to the offline version, but it still gives errors. Can you give me some help, please?
usage code:
Loading base Mistral model, along with custom code that enables bidirectional connections in decoder-only LLMs.
tokenizer = AutoTokenizer.from_pretrained( "McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp" ) config = AutoConfig.from_pretrained( "McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp", trust_remote_code=True ) model = AutoModel.from_pretrained( "McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp", trust_remote_code=True, config=config, torch_dtype=torch.bfloat16, device_map="cuda" if torch.cuda.is_available() else "cpu", )
Loading MNTP (Masked Next Token Prediction) model.
model = PeftModel.from_pretrained( model, "McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp", )
Modified code： local_base_model_path = "/home/McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp" tokenizer = AutoTokenizer.from_pretrained(local_base_model_path) config = AutoConfig.from_pretrained(local_base_model_path) model = AutoModel.from_pretrained(local_base_model_path, config=config,torch_dtype=torch.bfloat16, local_files_only=True) print(4)
errors: Traceback (most recent call last): File "/share/home/chenyuxuan/Llama3_8b_s.py", line 59, in
model = AutoModel.from_pretrained(local_model_path, config=config,torch_dtype=torch.bfloat16, local_files_only=True)
File "/share/home/chenyuxuan/.conda/envs/LLM2Vec/lib/python3.8/site-packages/transformers/models/auto/auto_factory.py", line 563, in from_pretrained
return model_class.from_pretrained(
File "/share/home/chenyuxuan/.conda/envs/LLM2Vec/lib/python3.8/site-packages/transformers/modeling_utils.py", line 3385, in from_pretrained
if has_file(pretrained_model_name_or_path, TF2_WEIGHTS_NAME, has_file_kwargs):
File "/share/home/chenyuxuan/.conda/envs/LLM2Vec/lib/python3.8/site-packages/transformers/utils/hub.py", line 627, in has_file
r = requests.head(url, headers=headers, allow_redirects=False, proxies=proxies, timeout=10)
File "/share/home/chenyuxuan/.conda/envs/LLM2Vec/lib/python3.8/site-packages/requests/api.py", line 100, in head
return request("head", url, kwargs)
File "/share/home/chenyuxuan/.conda/envs/LLM2Vec/lib/python3.8/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, kwargs)
File "/share/home/chenyuxuan/.conda/envs/LLM2Vec/lib/python3.8/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, send_kwargs)
File "/share/home/chenyuxuan/.conda/envs/LLM2Vec/lib/python3.8/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/share/home/chenyuxuan/.conda/envs/LLM2Vec/lib/python3.8/site-packages/requests/adapters.py", line 532, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)