Closed SH9959 closed 3 months ago
I'm not quite sure if it would be better to modify it like the following.😊
if tok.unk_token_id is None:
tok.unk_token_id = tok.pad_token_id
loss_mask = target_ids != tok.unk_token_id
Thank you very much for your advice; we have updated the code. Wishing you a pleasant life!
Line 118 of
ft_main.py
:The handling of tok.unk_token_id as None seems to be omitted here. If tok.unk_token_id is None, then on line 201 loss = -(torch.gather(probs, 1, target_ids) loss_mask).sum(1) / loss_mask.sum(1) will report an error of .sum()*