[BUG] 关于qwen2的bug报告

[macheng6]

datasets/utils.py文件中，关于qwen2的bug，QWen2Tokenizer 应该改为 Qwen2Tokenizer

def get_bos_eos_token_ids(tokenizer): if tokenizer.__class__.__name__ in [ 'QWenTokenizer', 'QWen2Tokenizer', 'Qwen2TokenizerFast' ]: bos_token_id = [] eos_token_id = tokenizer.eos_token_id assert eos_token_id is not None, \ 'Please set eos_token for Qwen tokenizer!' elif tokenizer.__class__.__name__ == 'ChatGLMTokenizer': bos_token_id = [64790, 64792] eos_token_id = tokenizer.eos_token_id else: bos_token_id = tokenizer.bos_token_id eos_token_id = tokenizer.eos_token_id if isinstance(bos_token_id, int): bos_token_id = [bos_token_id] if isinstance(eos_token_id, int): eos_token_id = [eos_token_id] return bos_token_id, eos_token_id

[hhaAndroid]

非常感谢你的反馈，确实有bug，你可以帮忙修复下吗？非常感谢

[macheng6]

抱歉，公司电脑不太方便