Open saeid93 opened 1 month ago
Can you try saving the phi-2 model without saving the phi-2 tokenizer, like this:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
torch.set_default_device("cuda")
model = AutoModelForCausalLM.from_pretrained("microsoft/phi-2", torch_dtype="auto", trust_remote_code=True)
model.save_pretrained("./saved_model")
Then try to compile it as you did last time.
@nicolasperez19 Thank you for your response but how is the model supposed to work without a tokenizer? I tried the code you mentioned and llamacpp is expecting the tokenizer to be there:
INFO:hf-to-gguf:Loading model: saved_model
INFO:gguf.gguf_writer:gguf: This GGUF file is for Little Endian only
INFO:hf-to-gguf:Set model parameters
INFO:hf-to-gguf:Set model tokenizer
Traceback (most recent call last):
File "/home/cc/polymorph/edge-setup/llama.cpp/convert-hf-to-gguf.py", line 2856, in <module>
main()
File "/home/cc/polymorph/edge-setup/llama.cpp/convert-hf-to-gguf.py", line 2841, in main
model_instance.set_vocab()
File "/home/cc/polymorph/edge-setup/llama.cpp/convert-hf-to-gguf.py", line 116, in set_vocab
self._set_vocab_gpt2()
File "/home/cc/polymorph/edge-setup/llama.cpp/convert-hf-to-gguf.py", line 502, in _set_vocab_gpt2
tokens, toktypes, tokpre = self.get_vocab_base()
File "/home/cc/polymorph/edge-setup/llama.cpp/convert-hf-to-gguf.py", line 377, in get_vocab_base
tokenizer = AutoTokenizer.from_pretrained(self.dir_model)
File "/home/cc/miniconda3/envs/peftenv/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 855, in from_pretrained
return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
File "/home/cc/miniconda3/envs/peftenv/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2070, in from_pretrained
raise EnvironmentError(
OSError: Can't load tokenizer for 'saved_model'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'saved_model' is the correct path to a directory containing all relevant files for a CodeGenTokenizerFast tokenizer.
What happened?
Despite phi-2 being listed here as a supported model, trying to compile it using the following command:
Fails with the following error:
This is also the code for saving the phi-2 model:
Name and Version
./main --version version: 3024 (2b737caa) built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu
What operating system are you seeing the problem on?
Linux
Relevant log output