Closed ZXP8080 closed 3 months ago
改一下模型载入部分就支持,直接复制cli multi gpu模型载入的部分进来
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Unused kwargs: ['_load_in_4bit', '_load_in_8bit', 'quant_method']. These kwargs are not used in <class 'transformers.utils.quantization_config.BitsAndBytesConfig'>.
low_cpu_mem_usage
was None, now set to True since model is quantized.
Traceback (most recent call last):
File "/home/liuhaodong/CogVLM2/basic_demo/cli_demo_multi_gpus.py", line 53, in
你好,我的报错,该怎么解决。单卡是没问题的
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. Unused kwargs: ['_load_in_4bit', '_load_in_8bit', 'quant_method']. These kwargs are not used in <class 'transformers.utils.quantization_config.BitsAndBytesConfig'>.
low_cpu_mem_usage
was None, now set to True since model is quantized. Traceback (most recent call last): File "/home/liuhaodong/CogVLM2/basic_demo/cli_demo_multi_gpus.py", line 53, in model = load_checkpoint_and_dispatch( File "/home/liuhaodong/python_venv/cogvlm/lib/python3.10/site-packages/accelerate/big_modeling.py", line 607, in load_checkpoint_and_dispatch load_checkpoint_in_model( File "/home/liuhaodong/python_venv/cogvlm/lib/python3.10/site-packages/accelerate/utils/modeling.py", line 1736, in load_checkpoint_in_model set_module_tensor_to_device( File "/home/liuhaodong/python_venv/cogvlm/lib/python3.10/site-packages/accelerate/utils/modeling.py", line 424, in set_module_tensor_to_device new_value = param_cls(new_value, requires_grad=old_value.requires_grad).to(device) File "/home/liuhaodong/python_venv/cogvlm/lib/python3.10/site-packages/bitsandbytes/nn/modules.py", line 324, in to return self._quantize(device) File "/home/liuhaodong/python_venv/cogvlm/lib/python3.10/site-packages/bitsandbytes/nn/modules.py", line 289, in _quantize w_4bit, quant_state = bnb.functional.quantize_4bit( File "/home/liuhaodong/python_venv/cogvlm/lib/python3.10/site-packages/bitsandbytes/functional.py", line 1234, in quantize_4bit raise ValueError(f"Blockwise quantization only supports 16/32-bit floats, but got {A.dtype}") ValueError: Blockwise quantization only supports 16/32-bit floats, but got torch.uint8你好,我的报错,该怎么解决。单卡是没问题的
int4版本的会报这个错,使用未量化加载成功,但报错:Traceback (most recent call last):
File "/home/liuhaodong/CogVLM2/basic_demo/cli_demo_multi_gpus.py", line 116, in
opanapi是否能够支持多卡推理