en-ashay
commented
5 months ago model=TheBloke_Llama-2-13B-GPTQ/model.safetensors, I also tried: Wizard-Vicuna-7B-Uncensored-GPTQ-4bit-128g.no-act-order.safetensors, same problem
Loading model ... --------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) Cell In[36], line 5 1 # MODEL_PATH = '/home/quang/working/LLMs/oobabooga_linux/text-generation-webui/models/TheBloke_Wizard-Vicuna-7B-Uncensored-GPTQ' 2 # CHECKPOINT_PATH = '/home/quang/working/LLMs/oobabooga_linux/text-generation-webui/models/TheBloke_Wizard-Vicuna-7B-Uncensored-GPTQ/Wizard-Vicuna-7B-Uncensored-GPTQ-4bit-128g.no-act-order.safetensors' 4 DEVICE = torch.device('cuda:0') ----> 5 model, tokenizer = load_model_main(MODEL_PATH, CHECKPOINT_PATH, DEVICE) File /mnt/data/generativeAgent_LLM/server/model.py:56, in load_model_main(model_para, checkpoint_para, device) 55 def load_model_main(model_para, checkpoint_para, device): ---> 56 model = load_quant(model_para, checkpoint_para, 4, 128) 57 model.to(device) 58 tokenizer = AutoTokenizer.from_pretrained(model_para) File /mnt/data/generativeAgent_LLM/server/model.py:47, in load_quant(model, checkpoint, wbits, groupsize) 45 if checkpoint.endswith('.safetensors'): 46 from safetensors.torch import load_file as safe_load ---> 47 model.load_state_dict(safe_load(checkpoint)) 48 else: 49 model.load_state_dict(torch.load(checkpoint)) File ~/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py:1671, in Module.load_state_dict(self, state_dict, strict) 1666 error_msgs.insert( 1667 0, 'Missing key(s) in state_dict: {}. '.format( 1668 ', '.join('"{}"'.format(k) for k in missing_keys))) 1670 if len(error_msgs) > 0: -> 1671 raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( 1672 self.__class__.__name__, "\n\t".join(error_msgs))) 1673 return _IncompatibleKeys(missing_keys, unexpected_keys) RuntimeError: Error(s) in loading state_dict for LlamaForCausalLM: Unexpected key(s) in state_dict: "model.layers.0.self_attn.rotary_emb.inv_freq", "model.layers.0.self_attn.k_proj.g_idx", "model.layers.0.self_attn.o_proj.g_idx", "model.layers.0.self_attn.q_proj.g_idx", "model.layers.0.self_attn.v_proj.g_idx", "model.layers.0.mlp.down_proj.g_idx", "model.layers.0.mlp.gate_proj.g_idx", "model.layers.0.mlp.up_proj.g_idx", "model.layers.1.self_attn.rotary_emb.inv_freq", "model.layers.1.self_attn.k_proj.g_idx", "model.layers.1.self_attn.o_proj.g_idx", "model.layers.1.self_attn.q_proj.g_idx", "model.layers.1.self_attn.v_proj.g_idx", "model.layers.1.mlp.down_proj.g_idx", "model.layers.1.mlp.gate_proj.g_idx", "model.layers.1.mlp.up_proj.g_idx", "model.layers.2.self_attn.rotary_emb.inv_freq", "model.layers.2.self_attn.k_proj.g_idx", "model.layers.2.self_attn.o_proj.g_idx", "model.layers.2.self_attn.q_proj.g_idx", "model.layers.2.self_attn.v_proj.g_idx", "model.layers.2.mlp.down_proj.g_idx", "model.layers.2.mlp.gate_proj.g_idx",
@myrainbowandsky @QuangBK I was also getting the same error but it was specific to this model, I later tried with microsoft phi 2 and mixtral and those two worked fine.
model=TheBloke_Llama-2-13B-GPTQ/model.safetensors, I also tried: Wizard-Vicuna-7B-Uncensored-GPTQ-4bit-128g.no-act-order.safetensors, same problem