RuntimeError: mat1 and mat2 must have the same dtype, but got Float and Half

[ulrika2cyl]

Base Model : InternLM-xcomposer2-vl-7b My Code : `import os

os.environ['CUDA_VISIBLE_DEVICES'] = '0'

import torch import json from tqdm import tqdm from modelscope import AutoTokenizer, AutoModelForCausalLM

ckpt_path = "/content-insight/base_model/internlm-xcomposer2-vl-7b" tokenizer = AutoTokenizer.from_pretrained(ckpt_path, revision='master', trust_remote_code=True)

torch_dtype=torch.float16 可以令模型以 float16 精度加载，否则 transformers 会将模型加载为 float32，导致显存不足

model = AutoModelForCausalLM.from_pretrained(ckpt_path, revision='master', torch_dtype=torch.float16, trust_remote_code=True,device_map="auto") model = model.eval() from modelscope import snapshot_download

self.vision_tower_name = snapshot_download("AI-ModelScope/clip-vit-large-patch14-336")

model.tokenizer = tokenizer

if name == 'main': text = '任务：你需要理解图片内容，理解图片中哪些文字是标题，并根据标题、相应的分类、其他重要信息对图片的内容以结构化的方式详细描述。 要求：1.忽略水印信息。2.只总结在图片上出现过的信息，不做发散。3.不要遗漏其他重要信息。' response, history = model.chat(query=text, image='体检有结节会癌变吗？一文读懂三种常见结节_1_蚂蚁保-重疾险_来自小红书网页版.jpg', tokenizer= tokenizer,history=[]) result_json.append({'picture_path':file_local_path, 'predict_result':response})

with open('/content-insight/234678/code/internlm_test/picture_analyze_0522_qwen_result_1752.txt', 'w') as file:
    json.dump(result_json, file, indent=4, ensure_ascii=False)`

The Error Detail: Traceback (most recent call last): File "/content-insight/234678/code/internlm_test/infer.py", line 33, in response, history = model.chat(query=text, image=file_local_path, tokenizer= tokenizer,history=[]) File "/opt/conda/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, kwargs) File "/root/.cache/huggingface/modules/transformers_modules/internlm-xcomposer2-vl-7b/modeling_internlm_xcomposer2.py", line 500, in chat image = self.encode_img(image) File "/root/.cache/huggingface/modules/transformers_modules/internlm-xcomposer2-vl-7b/modeling_internlm_xcomposer2.py", line 118, in encode_img img_embeds, atts_img, img_target = self.img2emb(image) File "/root/.cache/huggingface/modules/transformers_modules/internlm-xcomposer2-vl-7b/modeling_internlm_xcomposer2.py", line 122, in img2emb img_embeds = self.vision_proj(self.vit(image.to(self.device))) File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(args, kwargs) File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/container.py", line 215, in forward input = module(input) File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(args, **kwargs) File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 114, in forward return F.linear(input, self.weight, self.bias) RuntimeError: mat1 and mat2 must have the same dtype, but got Float and Half

我基于modelscope的代码进行了一点输入图片和文本的修改，然后遇到了这个问题，请问应该怎么解决

[ulrika2cyl]

补充一些信息： torch版本：2.1.0+cu121 transformers版本：4.31.0