Closed lldhliu closed 8 months ago
@hiyouga 大佬 我已经更新了tokenization_chatglm.py, 问题还没解决, 我的 transformers版本按照咱的最新代码要求进行了升级, 我查下来好像是transformers的版本问题, 烦请再帮忙看看
你更新后的文件长什么样?
@hiyouga 大佬 已解决, 解决方案如下: tokenization_chatglm.py代码中
class ChatGLMTokenizer(PreTrainedTokenizer):
vocab_files_names = {"vocab_file": "tokenizer.model"}
model_input_names = ["input_ids", "attention_mask", "position_ids"]
def __init__(self, vocab_file, padding_side="left", clean_up_tokenization_spaces=False, **kwargs):
super().__init__(padding_side=padding_side, clean_up_tokenization_spaces=clean_up_tokenization_spaces, **kwargs)
self.name = "GLMTokenizer"
self.vocab_file = vocab_file
self.tokenizer = SPTokenizer(vocab_file)
self.special_tokens = {
"<bos>": self.tokenizer.bos_id,
"<eos>": self.tokenizer.eos_id,
"<pad>": self.tokenizer.pad_id
}
其中 super().__init__(padding_side=padding_side, clean_up_tokenization_spaces=clean_up_tokenization_spaces, **kwargs) 移到下面去就好了:
class ChatGLMTokenizer(PreTrainedTokenizer):
vocab_files_names = {"vocab_file": "tokenizer.model"}
model_input_names = ["input_ids", "attention_mask", "position_ids"]
def __init__(self, vocab_file, padding_side="left", clean_up_tokenization_spaces=False, **kwargs):
self.name = "GLMTokenizer"
self.vocab_file = vocab_file
self.tokenizer = SPTokenizer(vocab_file)
self.special_tokens = {
"<bos>": self.tokenizer.bos_id,
"<eos>": self.tokenizer.eos_id,
"<pad>": self.tokenizer.pad_id
}
super().__init__(padding_side=padding_side, clean_up_tokenization_spaces=clean_up_tokenization_spaces, **kwargs)
@hiyouga 大佬 经过上面调整后可以运行了, 但聊天生成全部是 111111:
高级设置-模板选择 vanilla
@hiyouga 换了 vanilla 好像还不行
更新代码并使用 codegeex2 模板 a67a440644687dc2262134c0f2895f3ae42cae19 另外,CodeGeeX2 不是对话模型,只能提供续写功能,因此不会自动停止输出
Reminder
Reproduction
CUDA_VISIBLE_DEVICES=0 python src/train_web.py
Expected behavior
No response
System Info
No response
Others
No response