search

baichuan-inc / Baichuan2

A series of large language models developed by Baichuan Intelligent Technology
https://huggingface.co/baichuan-inc
Apache License 2.0
4.08k stars 293 forks source link

baichuan-7B-chat-4bits跑不通 #366

Open Yining0907 opened 8 months ago

Yining0907 commented 8 months ago

help，根据issues里下了bitsandbytes==0.41.1，但是还是报错Baichuan7BChatTextGeneration: 'NoneType' object is not subscriptable

sxndqc commented 8 months ago

好像pytorch_model里面存储的quant_state的类型错误了，应该是QuantState,但是从模型中加载出来的是一个List

Yining0907 commented 8 months ago

好像pytorch_model里面存储的quant_state的类型错误了，应该是QuantState,但是从模型中加载出来的是一个List

所以应该怎么修改呢？

sxndqc commented 7 months ago

@Yining0907 bitsandbytes 0.41.0是支持的，但是accelerate0.26.2最近一次更新出了点bug，我提了还没pull进去

qingchen177 commented 7 months ago

我的报这个错有大佬看看吗？ValueError: We need an offload_dir to dispatch this model according to this device_map, the following submodules need to be offloaded: model.layers.2, model.layers.3,..., model.norm, lm_head.