search

baichuan-inc / Baichuan2

A series of large language models developed by Baichuan Intelligent Technology
https://huggingface.co/baichuan-inc
Apache License 2.0
4.08k stars 292 forks source link

有地方可以下载可用的Baichuan2-13B-Chat-8bits吗？ #244

Open czhcc opened 11 months ago

czhcc commented 11 months ago

请问有地方可以下载可用的Baichuan2-13B-Chat-8bits吗？用官方提供的方法保存，加载会出错

jameswu2014 commented 11 months ago

save_pretrained保存的？加载报什么错？

czhcc commented 11 months ago

![Uploading 20231031173720.png…]() 不是加载，是保存时 AttributeError:BaichuanTokenizer' object has no attribute 'sp_model'

leoterry-ulrica commented 11 months ago

save_pretrained保存的？加载报什么错？ 加载baichuan2-13b-8bit量化模型加载后报错：

2023-10-31 21:01:51 | WARNING | accelerate.big_modeling | You shouldn't move a model when it is dispatched on multiple devices. 2023-10-31 21:01:51 | ERROR | stderr | Process model_worker - baichuan2-13b-chat-8bits: 2023-10-31 21:01:51 | ERROR | stderr | Traceback (most recent call last): 2023-10-31 21:01:51 | ERROR | stderr | File "/mnt/anaconda3/envs/python3.10/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap 2023-10-31 21:01:51 | ERROR | stderr | self.run() 2023-10-31 21:01:51 | ERROR | stderr | File "/mnt/anaconda3/envs/python3.10/lib/python3.10/multiprocessing/process.py", line 108, in run 2023-10-31 21:01:51 | ERROR | stderr | self._target(self._args, self._kwargs) 2023-10-31 21:01:51 | ERROR | stderr | File "/mnt/llm/.../startup.py", line 382, in run_model_worker 2023-10-31 21:01:51 | ERROR | stderr | app = create_model_worker_app(log_level=log_level, kwargs) 2023-10-31 21:01:51 | ERROR | stderr | File "/mnt/llm/.../startup.py", line 210, in create_model_worker_app 2023-10-31 21:01:51 | ERROR | stderr | worker = ModelWorker( 2023-10-31 21:01:51 | ERROR | stderr | File "/mnt/anaconda3/envs/python3.10/lib/python3.10/site-packages/fastchat/serve/model_worker.py", line 74, in init 2023-10-31 21:01:51 | ERROR | stderr | self.model, self.tokenizer = load_model( 2023-10-31 21:01:51 | ERROR | stderr | File "/mnt/anaconda3/envs/python3.10/lib/python3.10/site-packages/fastchat/model/model_adapter.py", line 308, in load_model 2023-10-31 21:01:51 | ERROR | stderr | model.to(device) 2023-10-31 21:01:51 | ERROR | stderr | File "/mnt/anaconda3/envs/python3.10/lib/python3.10/site-packages/accelerate/big_modeling.py", line 416, in wrapper 2023-10-31 21:01:51 | ERROR | stderr | return fn(args, **kwargs) 2023-10-31 21:01:51 | ERROR | stderr | File "/mnt/anaconda3/envs/python3.10/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2060, in to 2023-10-31 21:01:51 | ERROR | stderr | raise ValueError( 2023-10-31 21:01:51 | ERROR | stderr | ValueError: .to is not supported for 4-bit or 8-bit bitsandbytes models. Please use the model as it is, since the model has already been set to the correct devices and casted to the correct dtype.

jameswu2014 commented 11 months ago

This error maybe relevant to transformers version.I guess you are using the latest version, you can try it with 4.31.0.

leoterry-ulrica commented 11 months ago

This error maybe relevant to transformers version.I guess you are using the latest version, you can try it with 4.31.0.

并不是，是因为在加载模型的时候没设置load_8bit=True。

czhcc commented 11 months ago

但为什么用8bits的，也没见使用内存有下降？

guiniao commented 11 months ago

@czhcc ，你好，我离线int8量化时，报错 如下： /home/lproot/dl/huggingface/Baichuan2-main/model_int8.py:10 in │ │ │ │ 7 # Model saving: model_id is the original model directory, and quant8_saved_dir is the di │ │ 8 model = AutoModelForCausalLM.from_pretrained("./models", load_in_8bit=True, device_map=" │ │ 9 │ │ ❱ 10 model.save_pretrained("./models_int8") │ │ 11 │ │ │ │ /home/lproot/.local/lib/python3.8/site-packages/transformers/modeling_utils.py:1820 in │ │ save_pretrained │ │ │ │ 1817 │ │ weights_name = SAFE_WEIGHTS_NAME if safe_serialization else WEIGHTS_NAME │ │ 1818 │ │ weights_name = _add_variant(weights_name, variant) │ │ 1819 │ │ │ │ ❱ 1820 │ │ shards, index = shard_checkpoint(state_dict, max_shard_size=max_shard_size, weig │ │ 1821 │ │ │ │ 1822 │ │ # Clean the folder from a previous save │ │ 1823 │ │ for filename in os.listdir(save_directory): │ │ │ │ /home/lproot/.local/lib/python3.8/site-packages/transformers/modeling_utils.py:318 in │ │ shard_checkpoint │ │ │ │ 315 │ storage_id_to_block = {} │ │ 316 │ │ │ 317 │ for key, weight in state_dict.items(): │ │ ❱ 318 │ │ storage_id = id_tensor_storage(weight) │ │ 319 │ │ │ │ 320 │ │ # If a weight shares the same underlying storage as another tensor, we put `we │ │ 321 │ │ if storage_id in storage_id_to_block: │ │ │ │ /home/lproot/.local/lib/python3.8/site-packages/transformers/pytorch_utils.py:290 in │ │ id_tensor_storage │ │ │ │ 287 │ guaranteed to be unique and constant for this tensor's storage during its lifetime. │ │ 288 │ non-overlapping lifetimes may have the same id. │ │ 289 │ """ │ │ ❱ 290 │ return tensor.device, storage_ptr(tensor), storage_size(tensor) │ │ 291 │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ AttributeError: 'str' object has no attribute 'device'

请问你遇到过这个问题吗，transformers版本时4.30.0，torch版本2.0

FewBox commented 11 months ago

transformers版本问题，换transformers==4.30.0 torch==2.1.0+cu118。即使你使用了cuda12.1。兼容性不太好，我之前一直没问题，升级了下才出现问题。建议export出配置，防止升级的时候出问题。 百川的Requirement文件写的不好，如果你手动https://pytorch.org/get-started/locally/安装的话又会出现这样的问题，总之成功后多备份。

Gou-dan-Li commented 10 months ago

@czhcc，你好，我离线int8量化时，报错 如下： /home/lproot/dl/huggingface/Baichuan2-main/model_int8.py:10 in │ │ │ │ 7 # 模型保存：model_id为原始模型目录，quant8_saved_dir是 di │ │ 8 model = AutoModelForCausalLM.from_pretrained("./models", load_in_8bit=True, device_map=" │ │ 9 │ │ ❱ 10 model.save_pretrained("./models_int8") │ │ 11 │ │ │ │ / home/lproot/.local/lib/python3.8/site-packages/transformers/modeling_utils.py:1820 in │ │ save_pretrained │ │ │ │ 1817 │ │ Weights_name = SAFE_WEIGHTS_NAME if safe_serialization else WEIGHTS_NAME │ │ 1818 │ │weights_name = _add_variant (weights_name,variant) │ │ 1819 │ │ │ │ ❱ 1820 │ │ shards, index = shard_checkpoint(state_dict, max_shard_size=max_shard_size, weig │ │ 1821 │ │ │ │ 1822 │ │ # 清除之前保存的文件夹 │ │ 1823 │ │ os.listdir(save_directory) 中的文件名： │ │ │ │ /home/lproot/.local/lib/python3.8/site-packages/transformers/modeling_utils.py:318 in │ │ shard_checkpoint │ │ │ │ 315 │ storage_id_to_block = {} │ │ 316 │ │ │ 317 │ 对于 key，state_dict.items() 中的权重： │ │ ❱ 318 │ │ storage_id = id_tensor_storage(weight) │ │ 319 │ │ │ │ 320 │ │ # 如果 aweight与另一个张量共享相同的底层存储，我们将`we │ │ 321 │ │ if storage_id in storage_id_to_block: │ │ │ │ /home/lproot/.local/lib/python3.8/site-packages/transformers/pytorch_utils.py:290 in │ │ id_tensor_storage │ │ │ │ 287 │ 保证该张量的存储在其生命周期内是唯一且恒定的。│ │ 288 │ 非重叠的生命周期可以具有相同的 id。│ │ 289 │ “”” │ │ ❱ 290 │ 返回tensor.device, storage_ptr(张量), storage_size(张量) │ │ 291 │ ╰────────────────── ──────────────────────────────────────────────────── ──────────────────────────────╯ AttributeError: 'str' 对象没有属性 'device'

请问你遇到过这个问题吗，transformers版本时4.30.0，torch版本2.0

别保存，加载完直接干就行