search

yuanzhoulvpi2017 / zero_nlp

中文nlp解决方案(大模型、数据、模型、训练、推理)
MIT License
3.03k stars 368 forks source link

AttributeError: 'ChatGLMTokenizer' object has no attribute 'eop_token_id' #111

Closed withyou971 closed 1 year ago

withyou971 commented 1 year ago

ChatGLMTokenizer(name_or_path='model/chatglm-6b', vocab_size=130344, model_max_length=1000000000000000019884624838656, is_fast=False, padding_side='left', truncation_side='right', special_tokens={'bos_token': '', 'eos_token': '', 'unk_token': '', 'pad_token': '', 'mask_token': '[MASK]'}) 这个是python版本不对吗，python3.10.6

Frank-bpm commented 1 year ago

我也发现了这个问题，引用其他地方的回答：

Frank-bpm commented 1 year ago

注意model_name_or_path的模型，要用我提供的模型

Frank-bpm commented 1 year ago

训练lora代码报错 AttributeError: 'ChatGLMTokenizer' object has no attribute 'eop_token_id' [tokenizer.eop_token_id] 是不是应该改成 [tokenizer.sp_tokenizer[tokenizer.eos_token]]

yuanzhoulvpi2017 commented 1 year ago

训练lora代码报错 AttributeError: 'ChatGLMTokenizer' object has no attribute 'eop_token_id' [tokenizer.eop_token_id] 是不是应该改成 [tokenizer.sp_tokenizer[tokenizer.eos_token]]

看清楚要求，使用的是chatglm6b-dddd。

withyou971 commented 1 year ago

感谢，我下载这个试试

PedroNeal commented 6 months ago

感谢，我下载这个试试 屏幕截图 2024-05-12 200400

请问是怎么解决的