RuntimeError: mat1 and mat2 shapes cannot be multiplied (164x4096 and 1x8388608)

[adaaaaaa]

python generate_4bit.py --model_path decapoda-research/llama-7b-hf --lora_path Facico/Chinese-Vicuna-lora-7b-3epoch-belle-and-guanaco --use_local 0 上面的命令报错了。。。

/home/nano/.local/lib/python3.10/site-packages/gradio/inputs.py:27: UserWarning: Usage of gradio.inputs is deprecated, and will not be supported in the future, please import your component from gradio.components warnings.warn( /home/nano/.local/lib/python3.10/site-packages/gradio/inputs.py:30: UserWarning: optional parameter is deprecated, and it has no effect super().init( /home/nano/.local/lib/python3.10/site-packages/gradio/inputs.py:30: UserWarning: numeric parameter is deprecated, and it has no effect super().init( Running on local URL: http://127.0.0.1:7860 Traceback (most recent call last): File "/home/nano/.local/lib/python3.10/site-packages/gradio/routes.py", line 437, in run_predict output = await app.get_blocks().process_api( File "/home/nano/.local/lib/python3.10/site-packages/gradio/blocks.py", line 1346, in process_api result = await self.call_function( File "/home/nano/.local/lib/python3.10/site-packages/gradio/blocks.py", line 1090, in call_function prediction = await utils.async_iteration(iterator) File "/home/nano/.local/lib/python3.10/site-packages/gradio/utils.py", line 341, in async_iteration return await iterator.anext() File "/home/nano/.local/lib/python3.10/site-packages/gradio/interface.py", line 633, in fn async for output in iterator: File "/home/nano/.local/lib/python3.10/site-packages/gradio/utils.py", line 334, in anext return await anyio.to_thread.run_sync( File "/home/nano/.local/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "/home/nano/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread return await future File "/home/nano/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run result = context.run(func, args) File "/home/nano/.local/lib/python3.10/site-packages/gradio/utils.py", line 317, in run_sync_iterator_async return next(iterator) File "/data/Chinese-Vicuna/generate_4bit.py", line 152, in evaluate for generation_output in model.stream_generate( File "/data/Chinese-Vicuna/utils.py", line 657, in stream_beam_search outputs = self( File "/home/nano/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl return forward_call(input, kwargs) File "/home/nano/.local/lib/python3.10/site-packages/peft/peft_model.py", line 678, in forward return self.base_model( File "/home/nano/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl return forward_call(*input, *kwargs) File "/home/nano/.local/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(args, kwargs) File "/home/nano/.local/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 688, in forward outputs = self.model( File "/home/nano/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl return forward_call(*input, kwargs) File "/home/nano/.local/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(*args, *kwargs) File "/home/nano/.local/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 578, in forward layer_outputs = decoder_layer( File "/home/nano/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl return forward_call(input, kwargs) File "/home/nano/.local/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(*args, kwargs) File "/home/nano/.local/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 292, in forward hidden_states, self_attn_weights, present_key_value = self.self_attn( File "/home/nano/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl return forward_call(*input, *kwargs) File "/home/nano/.local/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(args, kwargs) File "/home/nano/.local/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 194, in forward query_states = self.q_proj(hidden_states).view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2) File "/home/nano/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl return forward_call(*input, **kwargs) File "/home/nano/.local/lib/python3.10/site-packages/peft/tuners/lora.py", line 565, in forward result = F.linear(x, transpose(self.weight, self.fan_in_fan_out), bias=self.bias) RuntimeError: mat1 and mat2 shapes cannot be multiplied (164x4096 and 1x8388608)

[LGAG]

我也遇到了这样的问题，但是是在用finetune_4bit.py的时候出现的

[Facico]

把依赖对齐：https://github.com/Facico/Chinese-Vicuna/blob/master/requirements_4bit.txt

[18065013]

我在双卡3090正常finetune非4bit下也存在该问题，不过我两张3090非同一厂商的，求解