Workaround for error when starting TheBloke/WizardLM-30B-Uncensored-GPTQ

Hi! First of all, great job on this! Running the 13B model with your repo and LangChain works great for me!

There's now a 30B GPTQ model, and I can start it in Oobabooga by just not specifying --groupsize 128.

As a note for anybody trying the 30B model with this repo/workaround: Commenting out this line allows it to start.

For anybody searching the error message, this is what I get:

Traceback (most recent call last):
  File "/home/tf/abhinand5/gptq_for_langchain/demo.py", line 84, in <module>
    llm_chain = create_gptq_llm_chain(mem_buf)
  File "/home/tf/abhinand5/gptq_for_langchain/demo.py", line 28, in create_gptq_llm_chain
    model, tokenizer = load_quantized_model(MODEL_NAME, args=AttributeDict(args))
  File "/home/tf/abhinand5/gptq_for_langchain/wizardlm_langchain/model.py", line 107, in load_quantized_model
    model = _load_quant(
  File "/home/tf/abhinand5/gptq_for_langchain/wizardlm_langchain/model.py", line 83, in _load_quant
    model.load_state_dict(safe_load(checkpoint), strict=False)
  File "/home/tf/.conda/envs/wizardlm_langchain/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2041, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for LlamaForCausalLM:
        size mismatch for model.layers.0.self_attn.k_proj.qzeros: copying a param with shape torch.Size([1, 832]) from checkpoint, the shape in current model is torch.Size([52, 832]).
        size mismatch for model.layers.0.self_attn.k_proj.scales: copying a param with shape torch.Size([1, 6656]) from checkpoint, the shape in current model is torch.Size([52, 6656]).
        size mismatch for model.layers.0.self_attn.o_proj.qzeros: copying a param with shape torch.Size([1, 832]) from checkpoint, the shape in current model is torch.Size([52, 832]).
        size mismatch for model.layers.0.self_attn.o_proj.scales: copying a param with shape torch.Size([1, 6656]) from checkpoint, the shape in current model is torch.Size([52, 6656]).
        size mismatch for model.layers.0.self_attn.q_proj.qzeros: copying a param with shape torch.Size([1, 832]) from checkpoint, the shape in current model is torch.Size([52, 832]).
        size mismatch for model.layers.0.self_attn.q_proj.scales: copying a param with shape torch.Size([1, 6656]) from checkpoint, the shape in current model is torch.Size([52, 6656]).
...
...
...
        size mismatch for model.layers.59.mlp.gate_proj.scales: copying a param with shape torch.Size([1, 17920]) from checkpoint, the shape in current model is torch.Size([52, 17920]).
        size mismatch for model.layers.59.mlp.up_proj.qzeros: copying a param with shape torch.Size([1, 2240]) from checkpoint, the shape in current model is torch.Size([52, 2240]).
        size mismatch for model.layers.59.mlp.up_proj.scales: copying a param with shape torch.Size([1, 17920]) from checkpoint, the shape in current model is torch.Size([52, 17920]).

abhinand5 / gptq_for_langchain

Workaround for error when starting TheBloke/WizardLM-30B-Uncensored-GPTQ #1