search

wangzhaode / llm-export

llm-export can export llm model to onnx.
Apache License 2.0
187 stars 21 forks source link

[Error]: Error Converting Llama 2 7B model #16

Closed Nick-infinity closed 7 months ago

Nick-infinity commented 7 months ago

Hello, I am trying to convert llama 2 7B model to mnn. I am using all the pip packages version as mentioned in requirements.txt with MNN version 2.8.1

I am getting the below error while converting the model 

 File "/Users/.cache/huggingface/modules/transformers_modules/Llama2_7B/modeling_llama.py", line 322, in forward
    key_states = torch.cat([past_key_value[0], key_states], dim=2)
RuntimeError: Sizes of tensors must match except in dimension 2. Expected size 32 but got size 4 for tensor number 1 in the list.

This error is triggered at this step https://github.com/wangzhaode/llm-export/blob/25fa2b033957d891691400c92be0ddacdde82c56/llm_export.py#L1013

The value of self.past_kv_shape = [32, 2, 1, 32, 0, 128] in Llama2_7b_Chat class

This code https://github.com/wangzhaode/llm-export/blob/25fa2b033957d891691400c92be0ddacdde82c56/llm_export.py#L256 Results in [2, 1, 32, 0, 128] as past_key_values and past_key_values[0] is indeed 2 from this logic used in modelling.py

How did you fix this issue?

Nick-infinity commented 7 months ago

Please ignore this issue as of now. I did a mismatch while adding custom llama 7b model. I will reopen it if it doesnt work