Closed xionghao132 closed 1 month ago
response = tokenizer(f"Assistant: {example['output']}<|end▁of▁sentence|>", add_special_tokens=False) input_ids = instruction["input_ids"] + response["input_ids"] + [tokenizer.pad_token_id] attention_mask = instruction["attention_mask"] + response["attention_mask"] + [1] # 因为eos token咱们也是要关注的所以 补充为1 labels = [-100] * len(instruction["input_ids"]) + response["input_ids"] + [tokenizer.pad_token_id] 想问一下,<|end▁of▁sentence|>是否多余,因为后面加了一个tokenizer.pad_token_id,也表示<|end▁of▁sentence|>
这个应该无所谓吧?因为都是pad_token
好的谢谢解答
response = tokenizer(f"Assistant: {example['output']}<|end▁of▁sentence|>", add_special_tokens=False) input_ids = instruction["input_ids"] + response["input_ids"] + [tokenizer.pad_token_id] attention_mask = instruction["attention_mask"] + response["attention_mask"] + [1] # 因为eos token咱们也是要关注的所以 补充为1 labels = [-100] * len(instruction["input_ids"]) + response["input_ids"] + [tokenizer.pad_token_id]
想问一下,<|end▁of▁sentence|>是否多余,因为后面加了一个tokenizer.pad_token_id,也表示<|end▁of▁sentence|>