Failed to disable ExLlama on pretty much any model

[Jacoby1218]

Describe the bug

attempting to load a model results in the error ERROR:Failed to disable exllama. Does the config.json for this model contain the necessary quantization info?

Is there an existing issue for this?

• [X] I have searched the existing issues

Reproduction

Load a model.

Screenshot

No response

Logs

2023-12-18 10:50:49 ERROR:Failed to disable exllama. Does the config.json for this model contain the necessary quantization info?
Traceback (most recent call last):
  File "D:\oobabooga_windows\oobabooga_windows\text-generation-webui\modules\models.py", line 227, in huggingface_loader
    bits=config.quantization_config.get('bits', 4),
         ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\oobabooga_windows\oobabooga_windows\text-generation-webui\installer_files\env\Lib\site-packages\transformers\configuration_utils.py", line 265, in __getattribute__
    return super().__getattribute__(key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'LlamaConfig' object has no attribute 'quantization_config'

Loading checkpoint shards:   0%|                                                                | 0/13 [00:01<?, ?it/s]
2023-12-18 10:50:52 ERROR:Failed to load the model.
Traceback (most recent call last):
  File "D:\oobabooga_windows\oobabooga_windows\text-generation-webui\modules\ui_model_menu.py", line 210, in load_model_wrapper
    shared.model, shared.tokenizer = load_model(selected_model, loader)
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\oobabooga_windows\oobabooga_windows\text-generation-webui\modules\models.py", line 89, in load_model
    output = load_func_map[loader](model_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\oobabooga_windows\oobabooga_windows\text-generation-webui\modules\models.py", line 244, in huggingface_loader
    model = LoaderClass.from_pretrained(path_to_model, **params)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\oobabooga_windows\oobabooga_windows\text-generation-webui\installer_files\env\Lib\site-packages\transformers\models\auto\auto_factory.py", line 566, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\oobabooga_windows\oobabooga_windows\text-generation-webui\installer_files\env\Lib\site-packages\transformers\modeling_utils.py", line 3694, in from_pretrained
    ) = cls._load_pretrained_model(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\oobabooga_windows\oobabooga_windows\text-generation-webui\installer_files\env\Lib\site-packages\transformers\modeling_utils.py", line 4104, in _load_pretrained_model
    new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model(
                                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\oobabooga_windows\oobabooga_windows\text-generation-webui\installer_files\env\Lib\site-packages\transformers\modeling_utils.py", line 778, in _load_state_dict_into_meta_model
    set_module_tensor_to_device(model, param_name, param_device, **set_module_kwargs)
  File "D:\oobabooga_windows\oobabooga_windows\text-generation-webui\installer_files\env\Lib\site-packages\accelerate\utils\modeling.py", line 317, in set_module_tensor_to_device
    new_value = value.to(device)
                ^^^^^^^^^^^^^^^^
  File "D:\oobabooga_windows\oobabooga_windows\text-generation-webui\installer_files\env\Lib\site-packages\torch\cuda\__init__.py", line 289, in _lazy_init
    raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

System Info

Windows 11 23H2
Intel ARC A770 16GB
32GB RAM

[VertexMachine]

That won't probably help much, but I just updated and I get the same error that CUDA is not enabled. What I noticed though is that after the last update when I run update_windows.bat script it installs everything in CPU only mode....

[VertexMachine]

This might help though, I deleted installer_files directory, run start_windows.bat again and it installed the correct version of torch (with CUDA) now.

[github-actions[bot]]

This issue has been closed due to inactivity for 6 weeks. If you believe it is still relevant, please leave a comment below. You can tag a developer in your comment.