Closed Gary-code closed 3 days ago
4090
x
new_input_ids = tokenizer.apply_chat_template([message], tokenize=True, return_dict=False)[2:] should be new_input_ids = tokenizer.apply_chat_template([message], tokenize=True, return_dict=False)[0][2:]?
new_input_ids = tokenizer.apply_chat_template([message], tokenize=True, return_dict=False)[2:]
new_input_ids = tokenizer.apply_chat_template([message], tokenize=True, return_dict=False)[0][2:]
Pull the latest hf file, chat_template has been placed in jinjia
System Info / 系統信息
4090
Who can help? / 谁可以帮助到您?
x
Information / 问题信息
Reproduction / 复现过程
new_input_ids = tokenizer.apply_chat_template([message], tokenize=True, return_dict=False)[2:]
should benew_input_ids = tokenizer.apply_chat_template([message], tokenize=True, return_dict=False)[0][2:]
?Expected behavior / 期待表现
new_input_ids = tokenizer.apply_chat_template([message], tokenize=True, return_dict=False)[2:]
should benew_input_ids = tokenizer.apply_chat_template([message], tokenize=True, return_dict=False)[0][2:]
?