微调之后模型加载不进去

[JerryPW]

我进行微调之后把pretrain地址换成了checkpoint中的地址，然后运行python cli_demo.py，会发现报出下面这样的问题：

模型使用Automodel进行加载的，请问为什么会报这样的错误呢？

[SixGoodX]

你好，你能运行这段代码吗？我前几天还可以运行，但是现在就不行

from transformers import AutoTokenizer, AutoModel import os

os.environ["CUDA_VISIBLE_DEVICES"] = "1" model_path = "./model/visualglm-6b" tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True) model = AutoModel.from_pretrained(model_path, trust_remote_code=True).quantize(4).half().cuda()

image_path = "user_1.png" response, history = model.chat(tokenizer, image_path, "描述这张图片。", history=[]) print(response)

[SixGoodX]

你可以借鉴这段代码， 加载checkpoint试试

CHECKPOINT_PATH = 'ptuning/output/mdd-chatglm-6b-int4-pt-128-2e-2/checkpoint-2000' config = AutoConfig.from_pretrained(model_path, trust_remote_code=True, pre_seq_len=128) model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True) prefix_state_dict = torch.load(os.path.join(CHECKPOINT_PATH, "pytorch_model.bin")) new_prefix_state_dict = {} for k, v in prefix_state_dict.items(): if k.startswith("transformer.prefix_encoder."): new_prefix_state_dict[k[len("transformer.prefix_encoder."):]] = v model.transformer.prefix_encoder.load_state_dict(new_prefix_state_dict)

model = model.quantize(4)

model = model.half().cuda() model.transformer.prefix_encoder.float()

[JerryPW]

你可以借鉴这段代码， 加载checkpoint试试 CHECKPOINT_PATH = 'ptuning/output/mdd-chatglm-6b-int4-pt-128-2e-2/checkpoint-2000' config = AutoConfig.from_pretrained(model_path, trust_remote_code=True, pre_seq_len=128) model = AutoModel.from_pretrained(model_path, config=config, trust_remote_code=True) prefix_state_dict = torch.load(os.path.join(CHECKPOINT_PATH, "pytorch_model.bin")) new_prefix_state_dict = {} for k, v in prefix_state_dict.items(): if k.startswith("transformer.prefix_encoder."): new_prefix_state_dict[k[len("transformer.prefix_encoder."):]] = v model.transformer.prefix_encoder.load_state_dict(new_prefix_state_dict)

model = model.quantize(4)

model = model.half().cuda() model.transformer.prefix_encoder.float()

这个是不是ChatGLM那个官方文档的微调指南？因为我看我的checkpoint里面没有关于transformer.prefix_encoder的网络层，而且刚才试了一下，会报错AttributeError: 'BaseTransformer' object has no attribute 'prefix_encoder。所以我估计不太行得通

[JerryPW]

from transformers import AutoTokenizer, AutoModel import os

os.environ["CUDA_VISIBLE_DEVICES"] = "1" model_path = "./model/visualglm-6b" tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True) model = AutoModel.from_pretrained(model_path, trust_remote_code=True).quantize(4).half().cuda()

image_path = "user_1.png" response, history = model.chat(tokenizer, image_path, "描述这张图片。", history=[]) print(response)

我运行了您的这段代码，会报以下这个错误，不知道和您一不一样： Traceback (most recent call last): File "/home/lihao/.local/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 259, in hf_raise_for_status response.raise_for_status() File "/usr/local/anaconda3/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://huggingface.co/visualglm-6b/resolve/main/tokenizer_config.json

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/lihao/.local/lib/python3.10/site-packages/transformers/utils/hub.py", line 417, in cached_file resolved_file = hf_hub_download( File "/home/lihao/.local/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn return fn(*args, *kwargs) File "/home/lihao/.local/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1195, in hf_hub_download metadata = get_hf_file_metadata( File "/home/lihao/.local/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn return fn(args, **kwargs) File "/home/lihao/.local/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1541, in get_hf_file_metadata hf_raise_for_status(r) File "/home/lihao/.local/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 291, in hf_raise_for_status raise RepositoryNotFoundError(message, response) from e huggingface_hub.utils._errors.RepositoryNotFoundError: 401 Client Error. (Request ID: Root=1-649e9656-08fd3ca02e889cbf5f63e8fb)

Repository Not Found for url: https://huggingface.co/visualglm-6b/resolve/main/tokenizer_config.json. Please make sure you specified the correct repo_id and repo_type. If you are trying to access a private or gated repo, make sure you are authenticated. Invalid username or password.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/lihao/VisualGLM-6B/test.py", line 6, in tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True) File "/home/lihao/.local/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 643, in from_pretrained tokenizer_config = get_tokenizer_config(pretrained_model_name_or_path, **kwargs) File "/home/lihao/.local/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 487, in get_tokenizer_config resolved_config_file = cached_file( File "/home/lihao/.local/lib/python3.10/site-packages/transformers/utils/hub.py", line 433, in cached_file raise EnvironmentError( OSError: visualglm-6b is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models' If this is a private repository, make sure to pass a token having permission to this repo with use_auth_token or log in with huggingface-cli login and pass use_auth_token=True.

[JerryPW]

问题大概是解决了，主要原因是在fintune的时候加入了8-bit量化，从而改变了model的结构，而cli_demo.py中是先加载模型再进行量化，所以会显示有新的层出现。解决方法目前是在base_model.py中加入 quantize()函数，但如何在不改变sat库的代码的前提下完成先量化在加载的方法，还未搞清。 以上仅供参考

[boh95]

@JerryPW 请问在base_model.py中加入 quantize()函数，具体在哪个地方加呢？感谢！

[JerryPW]

@Bo2Rim

在from_pretrain函数里面，在load_checkpoint之前加就可以