推理的时候找不到模型文件中pytorch_model.bin

[douyh]

我先从modelscope下载了模型（使用snapshot_download）在本地，然后使用下面代码推理

import torch
from modelscope import snapshot_download, AutoModel, AutoTokenizer

torch.set_grad_enabled(False)

# init model and tokenizer
model_dir = '/home/yuhao/.cache/modelscope/hub/Shanghai_AI_Laboratory/internlm-xcomposer2-7b-4bit'
model = AutoModel.from_pretrained(model_dir, trust_remote_code=True).to("cuda:1").eval()
tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)
model.tokenizer = tokenizer

text = '<ImageHere>Please describe this image in detail.'
image = 'road.jpg'
with torch.cuda.amp.autocast():
  response, _ = model.chat(tokenizer, query=text, image=image, history=[], do_sample=False)
print(response)

在本地推理的时候报错如下

OSError: Error no file named pytorch_model.bin, tf_model.h5, model.ckpt.index or flax_model.msgpack found

请问这个是什么原因呢、

[myownskyW7]

The inference code of 4bit models are different. Please check the ReadME here https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm-xcomposer2-7b-4bit/summary.