PROBLEM converting phi2, phi1.5, and phi1 models

[BBC-Esq]

I've tried converting the phi2/phi1.5/phi1 models and I get the following error when trying to convert all three models:

Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "D:\ctranslate2\Scripts\ct2-transformers-converter.exe\__main__.py", line 7, in <module>
  File "D:\ctranslate2\Lib\site-packages\ctranslate2\converters\transformers.py", line 2098, in main
    converter.convert_from_args(args)
  File "D:\ctranslate2\Lib\site-packages\ctranslate2\converters\converter.py", line 50, in convert_from_args
    return self.convert(
           ^^^^^^^^^^^^^
  File "D:\ctranslate2\Lib\site-packages\ctranslate2\converters\converter.py", line 89, in convert
    model_spec = self._load()
                 ^^^^^^^^^^^^
  File "D:\ctranslate2\Lib\site-packages\ctranslate2\converters\transformers.py", line 141, in _load
    spec = loader(model, tokenizer)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\ctranslate2\Lib\site-packages\ctranslate2\converters\transformers.py", line 193, in __call__
    spec = self.get_model_spec(model)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\ctranslate2\Lib\site-packages\ctranslate2\converters\transformers.py", line 1631, in get_model_spec
    num_layers=model.config.n_layer,
               ^^^^^^^^^^^^^^^^^^^^
  File "D:\ctranslate2\Lib\site-packages\transformers\configuration_utils.py", line 265, in __getattribute__
    return super().__getattribute__(key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'PhiConfig' object has no attribute 'n_layer'

And here is the command I'm using:

ct2-transformers-converter --model "D:/ctranslate2/phi-2" --output_dir "D:/ctranslate2\phi-2-ct2-float16" --quantization float16 --low_cpu_mem_usage --trust_remote_code --copy_files ".gitattributes" "added_tokens.json" "CODE_OF_CONDUCT.md" "configuration_phi.py" "generation_config.json" "LICENSE" "merges.txt" "model.safetensors.index.json" "modeling_phi.py" "NOTICE.md" "README.md" "SECURITY.md" "special_tokens_map.json" "tokenizer.json" "tokenizer_config.json" "vocab.json"

Anyone have any idea about this? Apparently, someone else has successfully converted it but they're using the source code from ctranslate2 directly. See the sample conversion code snippet at the bottom of this repo...Not sure why I can't do it with ct2-converter...

https://huggingface.co/michaelfeil/ct2fast-phi-1_5

[vince62s]

Microsoft has changed the config / layers naming in January, ct2 was never updated with this.

https://huggingface.co/microsoft/phi-2/commit/cb2f4533604d8b67de604e7df03bfe6f3ca22869#d2h-639018

[BBC-Esq]

This might be a good opportunity for to try my hand at a pull request with ctranslate2 even though I know zero about C++...but it seems relatively simple to modify the converter for phi...

[BBC-Esq]

For the exactly 2-3 other people in the universe who care about this ;-) here's a link where you can download the older version, but just fyi you have to download the files individually, you can't clone:

https://huggingface.co/microsoft/phi-2/tree/f6ba9c82af0f69e735470b457751c2726e3bbe3c

[BBC-Esq]

Last comment for me for the day...but here's what I get when trying to load the model with the sample llama2 script chat.py that guillikam made:

Traceback (most recent call last):
  File "D:\ctranslate2\chat.py", line 93, in <module>
    main()
  File "D:\ctranslate2\chat.py", line 10, in main
    sp = spm.SentencePieceProcessor(os.path.join(model_dir, "tokenizer.model"))
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\ctranslate2\Lib\site-packages\sentencepiece\__init__.py", line 468, in Init
    self.Load(model_file=model_file, model_proto=model_proto)
  File "D:\ctranslate2\Lib\site-packages\sentencepiece\__init__.py", line 961, in Load
    return self.LoadFromFile(model_file)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\ctranslate2\Lib\site-packages\sentencepiece\__init__.py", line 316, in LoadFromFile
    return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: Not found: "phi-2-ct2-float16\tokenizer.model": No such file or directory Error #2

It seems I was able to convert phi-2, but for whatever reason it didn't create the tokenizer.model in the new folder like it does when converting Mistral and other models...

[edit] I just realized that is way above my experience level...but maybe this'll help??

https://chat.openai.com/share/84e8c2a0-fab5-41d7-a84a-7f195d2f25f2

[minhthuc2502]

when convert the model, you can add this option to save the tokenizer.model in your folder: --copy_files tokenizer.model

[BBC-Esq]

Thanks @minhthuc2502 , but the repository doesn't seem to have a tokenizer.model, which is part of my confusion. BTW, I communicated with the guy who originally did the phi loader/converter, Michael Feil, and he said he doesn't have time to update it to adapt to the changes that Microsoft made to the model in January, 2024. I'd love to try but have no way of visualizing a model's architecture (Netron is mediocre for Pytorch models), and have zero c++ experience. Do you have a sense of if/when the phi loader/converter will be updated by you or someone else? Not a huge deal, just wondering. Anyways, here's where it says no tokenizer.model...I noticed that Michael Feil's ctranslate2 conversion of Phi 1.5 (before Microsoft's change) doesn't have that file...

[minhthuc2502]

It seems like you can't use sentenpiece in chat.py to tokenize the input. You can directly use tokenizer of transformers like: tokenizer = AutoTokenizer.from_pretrained(target_model, trust_remote_code=True).

We have no plans to do an update right now but it would be done if there is time.

[BBC-Esq]

I withdraw this "issue" because phi-2 is too difficult and when it does work, the results are poor...HOWEVER, I strongly recommend updating the scope "StableLM" models that ctranslate2 supports. The new Zephyr is awesome, for example, but there are a slew of other small 3b models that StableLM makes that are best for their sizes IMHO.