[Bug] RuntimeError: Tensors must have same number of dimensions: got 3 and 4

[NanoCode012]

Hello, thank you for the repo.

I attempted to run the bot with the same prompt as the example (as well as others), but got the error below. I was wondering if you have any ideas?

Traceback (most recent call last):
  File "/disc/bot/actions/__init__.py", line 140, in run_prompt
    output = await context.llama_engine.predict_text(prompt, # type: ignore
  File "/disc/bot/llama_model/engine.py", line 87, in predict_text
    generated_ids = self.model.generate(
  File "/usr/local/lib/python3.10/dist-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/disc/transformers/src/transformers/generation/utils.py", line 1452, in generate
    return self.sample(
  File "/disc/transformers/src/transformers/generation/utils.py", line 2468, in sample
    outputs = self(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/disc/transformers/src/transformers/models/llama/modeling_llama.py", line 772, in forward
    outputs = self.model(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/disc/transformers/src/transformers/models/llama/modeling_llama.py", line 621, in forward
    layer_outputs = decoder_layer(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/disc/transformers/src/transformers/models/llama/modeling_llama.py", line 318, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/disc/transformers/src/transformers/models/llama/modeling_llama.py", line 228, in forward
    query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin, offset=offset)
  File "/disc/transformers/src/transformers/models/llama/modeling_llama.py", line 142, in apply_rotary_pos_emb
    q_embed = (q * cos) + (rotate_half(q) * sin)
  File "/disc/transformers/src/transformers/models/llama/modeling_llama.py", line 136, in rotate_half
    return torch.cat((-x2, x1), dim=-1)
RuntimeError: Tensors must have same number of dimensions: got 3 and 4

Reproduce:

1. Use nvidia/cuda version 11.7 docker container
2. Setup python 3.10
3. Run commands following Readme
4. Start bot
5. Give it prompt
6. Errors

Additional context:

I tried to clone the latest https://github.com/qwopqwop200/GPTQ-for-LLaMa, copied the current engine.py from this repo, and edited gptq.py import a bit for quant.py. The issue persists.

[AmericanPresidentJimmyCarter]

I wonder if it's due to a push on the llama branch, try git checkout 68d640f7c368bcaaaecfc678f11908ebbd3d6176 in the transformers repo, then reinstall with pip install -e ..

[NanoCode012]

I wonder if it's due to a push on the llama branch, try git checkout 68d640f7c368bcaaaecfc678f11908ebbd3d6176 in the transformers repo, then reinstall with pip install -e ..

Hello, thank you @AmericanPresidentJimmyCarter . This solved the issue.

[AmericanPresidentJimmyCarter]

Thanks, I have updated the README.