search

oobabooga / text-generation-webui

A Gradio web UI for Large Language Models.
GNU Affero General Public License v3.0
40.84k stars 5.34k forks source link

Can't load awq model #6460

Open nNote1377 opened 1 month ago

nNote1377 commented 1 month ago

Describe the bug

I installed text generation webui and downloaded the model(TheBloke_Yarn-Mistral-7B-128k-AWQ) and I can't run it. I chose Transofmer as Model loader. I tried installing autoawq but the installed version of torch is not compatible with it, so while installation it downloads an older version but it still doesn’t work.

Is there an existing issue for this?

Reproduction

Select a model and try to load it

Screenshot

image image

Logs

C:\Users\barto\Documents\programy\ai\text-generation-webui\installer_files\env\Lib\site-packages\transformers\quantizers\auto.py:182: UserWarning: You passed `quantization_config` or equivalent parameters to `from_pretrained` but the model you're loading already has a `quantization_config` attribute. The `quantization_config` from the model will be used.However, loading attributes (e.g. ['use_cuda_fp16', 'use_exllama', 'max_input_length', 'exllama_config', 'disable_exllama']) will be overwritten with the one you passed to `from_pretrained`. The rest will be ignored.
  warnings.warn(warning_msg)
20:30:11-368425 ERROR    Failed to load the model.
Traceback (most recent call last):
  File "C:\Users\barto\Documents\programy\ai\text-generation-webui\modules\ui_model_menu.py", line 232, in load_model_wrapper
    shared.model, shared.tokenizer = load_model(selected_model, loader)
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\barto\Documents\programy\ai\text-generation-webui\modules\models.py", line 93, in load_model
    output = load_func_map[loader](model_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\barto\Documents\programy\ai\text-generation-webui\modules\models.py", line 263, in huggingface_loader
    model = LoaderClass.from_pretrained(path_to_model, **params)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\barto\Documents\programy\ai\text-generation-webui\installer_files\env\Lib\site-packages\transformers\models\auto\auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\barto\Documents\programy\ai\text-generation-webui\installer_files\env\Lib\site-packages\transformers\modeling_utils.py", line 3452, in from_pretrained
    hf_quantizer.validate_environment(
  File "C:\Users\barto\Documents\programy\ai\text-generation-webui\installer_files\env\Lib\site-packages\transformers\quantizers\quantizer_awq.py", line 53, in validate_environment
    raise ImportError("Loading an AWQ quantized model requires auto-awq library (`pip install autoawq`)")
ImportError: Loading an AWQ quantized model requires auto-awq library (`pip install autoawq`)
20:36:52-274133 INFO     Loading "TheBloke_Yarn-Mistral-7B-128k-AWQ"
20:36:52-281135 INFO     TRANSFORMERS_PARAMS=
{'low_cpu_mem_usage': True, 'torch_dtype': torch.float16}

20:36:52-285138 ERROR    Failed to load the model.
Traceback (most recent call last):
  File "C:\Users\barto\Documents\programy\ai\text-generation-webui\modules\ui_model_menu.py", line 232, in load_model_wrapper
    shared.model, shared.tokenizer = load_model(selected_model, loader)
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\barto\Documents\programy\ai\text-generation-webui\modules\models.py", line 93, in load_model
    output = load_func_map[loader](model_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\barto\Documents\programy\ai\text-generation-webui\modules\models.py", line 172, in huggingface_loader
    model = LoaderClass.from_pretrained(path_to_model, **params)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\barto\Documents\programy\ai\text-generation-webui\installer_files\env\Lib\site-packages\transformers\models\auto\auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\barto\Documents\programy\ai\text-generation-webui\installer_files\env\Lib\site-packages\transformers\modeling_utils.py", line 3452, in from_pretrained
    hf_quantizer.validate_environment(
  File "C:\Users\barto\Documents\programy\ai\text-generation-webui\installer_files\env\Lib\site-packages\transformers\quantizers\quantizer_awq.py", line 53, in validate_environment
    raise ImportError("Loading an AWQ quantized model requires auto-awq library (`pip install autoawq`)")
ImportError: Loading an AWQ quantized model requires auto-awq library (`pip install autoawq`)

System Info

CPU: AMD Ryzen 7 7800X3D
GPU: NVIDIA GeForce RTX 4070 Ti SUPER
DoctorsevenTOP commented 1 month ago

Describe the bug

I installed text generation webui and downloaded the model black-forest-labs_FLUX.1-schnell and some more, I have the same bug.

Is there an existing issue for this?

I have searched the existing issues

Reproduction

Select a model and try to load it image image

22:13:52-565871 ERROR Failed to load the model. OpenTrain-V2.safetensors: 4%|██▏ | 110M/2.43G [00:10<03:40, 11.3MB/s]Traceback (most recent call last): File "D:\Games\text-generation-webui-1.15\modules\ui_model_menu.py", line 232, in load_model_wrapper shared.model, shared.tokenizer = load_model(selected_model, loader) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\Games\text-generation-webui-1.15\modules\models.py", line 93, in load_model output = load_func_map[loader](model_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\Games\text-generation-webui-1.15\modules\models.py", line 155, in huggingface_loader config = AutoConfig.from_pretrained(path_to_model, trust_remote_code=shared.args.trust_remote_code) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\Games\text-generation-webui-1.15\installer_files\env\Lib\site-packages\transformers\models\auto\configuration_auto.py", line 1038, in from_pretrained raise ValueError( ValueError: Unrecognized model in models\stable-diffusion-v1-5_stable-diffusion-v1-5. Should have amodel_typekey in its config.json, or contain one of the following strings in its name: albert, align, altclip, audio-spectrogram-transformer, autoformer, bark, bart, beit, bert, bert-generation, big_bird, bigbird_pegasus, biogpt, bit, blenderbot, blenderbot-small, blip, blip-2, bloom, bridgetower, bros, camembert, canine, chameleon, chinese_clip, chinese_clip_vision_model, clap, clip, clip_text_model, clip_vision_model, clipseg, clvp, code_llama, codegen, cohere, conditional_detr, convbert, convnext, convnextv2, cpmant, ctrl, cvt, dac, data2vec-audio, data2vec-text, data2vec-vision, dbrx, deberta, deberta-v2, decision_transformer, deformable_detr, deit, depth_anything, deta, detr, dinat, dinov2, distilbert, donut-swin, dpr, dpt, efficientformer, efficientnet, electra, encodec, encoder-decoder, ernie, ernie_m, esm, falcon, falcon_mamba, fastspeech2_conformer, flaubert, flava, fnet, focalnet, fsmt, funnel, fuyu, gemma, gemma2, git, glpn, gpt-sw3, gpt2, gpt_bigcode, gpt_neo, gpt_neox, gpt_neox_japanese, gptj, gptsan-japanese, granite, granitemoe, graphormer, grounding-dino, groupvit, hiera, hubert, ibert, idefics, idefics2, imagegpt, informer, instructblip, instructblipvideo, jamba, jetmoe, jukebox, kosmos-2, layoutlm, layoutlmv2, layoutlmv3, led, levit, lilt, llama, llava, llava_next, llava_next_video, llava_onevision, longformer, longt5, luke, lxmert, m2m_100, mamba, mamba2, marian, markuplm, mask2former, maskformer, maskformer-swin, mbart, mctct, mega, megatron-bert, mgp-str, mimi, mistral, mixtral, mllama, mobilebert, mobilenet_v1, mobilenet_v2, mobilevit, mobilevitv2, mpnet, mpt, mra, mt5, musicgen, musicgen_melody, mvp, nat, nemotron, nezha, nllb-moe, nougat, nystromformer, olmo, olmoe, omdet-turbo, oneformer, open-llama, openai-gpt, opt, owlv2, owlvit, paligemma, patchtsmixer, patchtst, pegasus, pegasus_x, perceiver, persimmon, phi, phi3, pix2struct, pixtral, plbart, poolformer, pop2piano, prophetnet, pvt, pvt_v2, qdqbert, qwen2, qwen2_audio, qwen2_audio_encoder, qwen2_moe, qwen2_vl, rag, realm, recurrent_gemma, reformer, regnet, rembert, resnet, retribert, roberta, roberta-prelayernorm, roc_bert, roformer, rt_detr, rt_detr_resnet, rwkv, sam, seamless_m4t, seamless_m4t_v2, segformer, seggpt, sew, sew-d, siglip, siglip_vision_model, speech-encoder-decoder, speech_to_text, speech_to_text_2, speecht5, splinter, squeezebert, stablelm, starcoder2, superpoint, swiftformer, swin, swin2sr, swinv2, switch_transformers, t5, table-transformer, tapas, time_series_transformer, timesformer, timm_backbone, trajectory_transformer, transfo-xl, trocr, tvlt, tvp, udop, umt5, unispeech, unispeech-sat, univnet, upernet, van, video_llava, videomae, vilt, vipllava, vision-encoder-decoder, vision-text-dual-encoder, visual_bert, vit, vit_hybrid, vit_mae, vit_msn, vitdet, vitmatte, vits, vivit, wav2vec2, wav2vec2-bert, wav2vec2-conformer, wavlm, whisper, xclip, xglm, xlm, xlm-prophetnet, xlm-roberta, xlm-roberta-xl, xlnet, xmod, yolos, yoso, zoedepth

System info

CPU: AMD Ryzen 5 7500 GPU: NVIDIA GeForce RTX 4060 Ti

idhem963 commented 1 month ago

for nNote if you installed the same one as me (from this link https://github.com/oobabooga/text-generation-webui) you will find cmd_yourOS.bat/.sh (most likely cmd_windows.dat) open it and paste (pip install autoawq) hopefully it will fix your problem cuz auto awq is no longer supported and isnt downloaded automatically (plz refer to this sub this is where i found the solution https://www.reddit.com/r/Oobabooga/comments/1g86grc/i_get_an_error_when_i_choose_a_awq_model_need_help/?tl=fr :)

SergeantOsborne commented 1 day ago

same problem but also with Auto-GTQ. @idhem963 I added the pip install to the script and this doesn't work either, it tells me that it's already installed.