项目无法运行推理360Zhinao-7B-Chat-360K-Int4和360Zhinao-7B-Chat-32K-Int4两个量化版

Qihoo360 / 360zhinao

360zhinao

Apache License 2.0

278 stars 22 forks source link

项目无法运行推理360Zhinao-7B-Chat-360K-Int4和360Zhinao-7B-Chat-32K-Int4两个量化版 #7

Closed qingfeng2018 closed 5 months ago

qingfeng2018 commented 6 months ago

OSError: Error no file named pytorch_model.bin, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory /home/ai-models/360_model/360Zhinao-7B-Chat-360K-Int4. 运行模型推理报错，提示缺文件，但是魔搭模型文件夹内并没有这几个文件，运行非量化360Zhinao-7B-Chat-360K正常

gongxiaochun commented 6 months ago

请使用vllm运行，当前版本hf加载int4版本会报错。

qingfeng2018 commented 6 months ago

好的，谢谢