ChatGLM-6B模型导入问题

[Alternate-D]

作者您好！当我在前端输入框输入文本后，ChatGLM-6B模型推理时存在下面的问题：

INFO: Application startup complete. INFO: 127.0.0.1:33704 - "OPTIONS / HTTP/1.1" 200 OK Start generating... The dtype of attention mask (torch.int64) is not bool Traceback (most recent call last): File "/media/ln01/2t/usr/cw/ChatZoo/generator/chatbot.py", line 64, in chat output = self.generate(input_dict, gen_kwargs) File "/media/ln01/2t/usr/cw/ChatZoo/generator/transformersbot.py", line 69, in generate return self.model.generate(input_dict, gen_kwargs) File "/home/ln01/miniconda3/envs/ChatZoo/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, kwargs) File "/home/ln01/miniconda3/envs/ChatZoo/lib/python3.8/site-packages/transformers/generation/utils.py", line 1485, in generate return self.sample( File "/home/ln01/miniconda3/envs/ChatZoo/lib/python3.8/site-packages/transformers/generation/utils.py", line 2524, in sample outputs = self( File "/home/ln01/miniconda3/envs/ChatZoo/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "/home/ln01/miniconda3/envs/ChatZoo/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(args, kwargs) File "/media/ln01/2t/usr/cw/ChatZoo/generator/models/chatglm/modeling_chatglm.py", line 1178, in forward transformer_outputs = self.transformer( File "/home/ln01/miniconda3/envs/ChatZoo/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, kwargs) File "/media/ln01/2t/usr/cw/ChatZoo/generator/models/chatglm/modeling_chatglm.py", line 984, in forward layer_ret = layer( File "/home/ln01/miniconda3/envs/ChatZoo/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "/home/ln01/miniconda3/envs/ChatZoo/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(args, kwargs) File "/media/ln01/2t/usr/cw/ChatZoo/generator/models/chatglm/modeling_chatglm.py", line 624, in forward attention_input = self.input_layernorm(hidden_states) File "/home/ln01/miniconda3/envs/ChatZoo/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "/home/ln01/miniconda3/envs/ChatZoo/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(args, kwargs) File "/home/ln01/miniconda3/envs/ChatZoo/lib/python3.8/site-packages/torch/nn/modules/normalization.py", line 190, in forward return F.layer_norm( File "/home/ln01/miniconda3/envs/ChatZoo/lib/python3.8/site-packages/torch/nn/functional.py", line 2515, in layer_norm return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled) RuntimeError: expected scalar type Half but found Float**

请问您能帮助我吗？如果可以的话感激不尽！

[Alternate-D]

还有一点，我想知道你们的具体虚拟环境，麻烦您告诉我

[x54-729]

请问您是使用的最新的代码吗？chatglm的模型内部调用了Tensor.half()接口，所以对输入的类型有邀请，可以尝试一下启动时指定 --dtype=float32

[Alternate-D]

请问您是使用的最新的代码吗？chatglm的模型内部调用了Tensor.half()接口，所以对输入的类型有邀请，可以尝试一下启动时指定 --dtype=float32

1.我使用的是您的最新代码； 2.您的server.py文件中dtype默认就是float32 _parser.addargument( "--dtype", type=str, default="float32", help="Dtype to load model." ) 所以还是存在上述问题的

[Alternate-D]

请问您是使用的最新的代码吗？chatglm的模型内部调用了Tensor.half()接口，所以对输入的类型有邀请，可以尝试一下启动时指定 --dtype=float32

我已经解决这个问题了！根据您的提示，我在_transformerbot.py_的_load_model(self)_方法下进行了如下修改： _def load_model(self): """ Load model through transformers. """

mute warning

    trust_remote_code = issubclass(
        self.model_cls, _BaseAutoModelClass
    )
    # print(self.model_cls)
    # print(type(self.model_cls))
    # print(self.model_cls == ChatGLMForConditionalGeneration)
    self.model = self.model_cls.from_pretrained(
        self.model_name, torch_dtype=self.config.dtype,
        device_map="auto", trust_remote_code=trust_remote_code
    )
    if (self.model_cls == ChatGLMForConditionalGeneration):
        self.model = self.model.half()_

即：当导入的模型为ChatGLM时，手动把数据类型转换为half，希望您可以参考。 最后，感谢您百忙中抽出时间为我解答！

[x54-729]

请问您是使用的最新的代码吗？chatglm的模型内部调用了Tensor.half()接口，所以对输入的类型有邀请，可以尝试一下启动时指定 --dtype=float32

1.我使用的是您的最新代码； 2.您的server.py文件中dtype默认就是float32 _parser.addargument( "--dtype", type=str, default="float32", help="Dtype to load model." ) 所以还是存在上述问题的

不好意思一开始回复错了，应该是--dtype=float16 _(:з不过chatglm里存在一个assert语句，用float32应该会报错才对