FileNotFoundError: No safetensors found

[IamShubhamGupto]

Hello

I was trying out the mlx-vlm package and was able to run the default example here

However, replacing the model card with Efficient-Large-Model/VILA-13b-4bit-awq, it fails. heres the stack trace:

(mlx) shubham@Shubhams-MBP ~ % python -m mlx_vlm.generate --model Efficient-Large-Model/VILA-13b-4bit-awq \
--prompt "what are these?" --image "http://images.cocodataset.org/val2017/000000039769.jpg" \
--max-tokens 100 --temp 0.0

Fetching 7 files: 100%|████████████████████████| 7/7 [00:00<00:00, 14761.25it/s]
ERROR:root:No safetensors found in /Users/shubham/.cache/huggingface/hub/models--Efficient-Large-Model--VILA-13b-4bit-awq/snapshots/ab335be6c2a5b6a08d4784491bf270fe0ce7a41d
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/shubham/anaconda3/envs/mlx/lib/python3.12/site-packages/mlx_vlm/generate.py", line 107, in <module>
    main()
  File "/Users/shubham/anaconda3/envs/mlx/lib/python3.12/site-packages/mlx_vlm/generate.py", line 69, in main
    model, processor, image_processor, config = get_model_and_processors(args.model)
                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/shubham/anaconda3/envs/mlx/lib/python3.12/site-packages/mlx_vlm/generate.py", line 55, in get_model_and_processors
    model, processor = load(model_path, {"trust_remote_code": True})
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/shubham/anaconda3/envs/mlx/lib/python3.12/site-packages/mlx_vlm/utils.py", line 212, in load
    model = load_model(model_path, lazy)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/shubham/anaconda3/envs/mlx/lib/python3.12/site-packages/mlx_vlm/utils.py", line 123, in load_model
    raise FileNotFoundError(f"No safetensors found in {model_path}")
FileNotFoundError: No safetensors found in /Users/shubham/.cache/huggingface/hub/models--Efficient-Large-Model--VILA-13b-4bit-awq/snapshots/ab335be6c2a5b6a08d4784491bf270fe0ce7a41d
(mlx) shubham@Shubhams-MBP ~ %

I am using a m1 MacBook Pro. Im new to mlx-vlm but im happy to work on this to add support.

[Blaizzy]

Thanks for sharing!

This is happening because vila doesn't have any safetensors in the repo.

Here is what you can do:

Load the using transformers and torch, the save it:

# Load model directly
from transformers import AutoModelForCausalLM, AutoProcessor

model_id= "Efficient-Large-Model/VILA-7b"
model = AutoModelForCausalLM.from_pretrained(model_id)
processor = AutoProcessor.from_pretrained(model_id) 

model.save_pretrained("vila-7b")

processor.save_pretrained("vila-7b")

[Blaizzy]

The use the convert script and point the --hf-path to the saved folder

[IamShubhamGupto]

@Blaizzy thanks for the guide! I will try it out.

This seems like an automation we can add within mlx-vlm. Does it make sense for the library to do this by itself if a user provides a repo without safetensors?

[Blaizzy]

Most welcome!

Hard to say, because I don't want to add heavy dependencies (torch or TF) just to automate a rare use case.

I think for now, we can add this snippet in the error message.

Do you mind making a PR with this improved error message?

[IamShubhamGupto]

Thanks for sharing!

This is happening because vila doesn't have any safetensors in the repo.

Here is what you can do:

Load the using transformers and torch, the save it:

# Load model directly
from transformers import AutoModelForCausalLM, AutoProcessor

model_id= "Efficient-Large-Model/VILA-7b"
model = AutoModelForCausalLM.from_pretrained(model_id)
processor = AutoProcessor.from_pretrained(model_id) 

model.save_pretrained("vila-7b")

processor.save_pretrained("vila-7b")

im guessing this is a model specific error, another one might have worked

/Users/shubham/anaconda3/envs/mlx/lib/python3.12/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
Traceback (most recent call last):
  File "/Users/shubham/anaconda3/envs/mlx/lib/python3.12/site-packages/transformers/models/auto/configuration_auto.py", line 945, in from_pretrained
    config_class = CONFIG_MAPPING[config_dict["model_type"]]
                   ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/shubham/anaconda3/envs/mlx/lib/python3.12/site-packages/transformers/models/auto/configuration_auto.py", line 647, in __getitem__
    raise KeyError(key)
KeyError: 'llava_llama'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/shubham/Documents/workspace/forks/mlx-vlm/mlx_vlm/dummy.py", line 5, in <module>
    model = AutoModelForCausalLM.from_pretrained(model_id)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/shubham/anaconda3/envs/mlx/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 523, in from_pretrained
    config, kwargs = AutoConfig.from_pretrained(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/shubham/anaconda3/envs/mlx/lib/python3.12/site-packages/transformers/models/auto/configuration_auto.py", line 947, in from_pretrained
    raise ValueError(
ValueError: The checkpoint you are trying to load has model type `llava_llama` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.