[Bad Case]: 多模态 MiniCPM-V 推理报错

[c122-ode]

Description / 描述

代码： `import torch from PIL import Image from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained('openbmb/MiniCPM-V', trust_remote_code=True) tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V', trust_remote_code=True) model.eval().cuda()

image = Image.open('xx.jpg').convert('RGB') question = 'What is in the image?' msgs = [{'role': 'user', 'content': question}]

res, context, _ = model.chat( image=image, msgs=msgs, context=None, tokenizer=tokenizer, sampling=True, temperature=0.7 ) print(res)`

错误结果： .cache/huggingface/modules/transformers_modules/openbmb/MiniCPM-V-2/ee00ff7ce36667e7df81cb2a018951b663bdcc59/modeling_minicpm.py", line 284, in apply_rotary_pos_emb cos = cos[position_ids].unsqueeze(unsqueeze_dim) # [bs, 1, seq_len, dim] IndexError: index is out of bounds for dimension with size 0

Case Explaination / 案例解释

我发现MiniCPMV 中运行到_decode 时 self.llm.generate 的 inputs_embeds 输入是正常的 但是 MiniCPMForCausalLM forward 传入的 inputs_embeds 居然是None

[LDLINGLINGLING]

是否和transformer版本有关呢，建议升级transformer版本到4.40

[Komorebi1102]

是否和transformer版本有关呢，建议升级transformer版本到4.40

有用