Closed vivekmadan2 closed 2 weeks ago
Any support on this? We found out that the training is doing just next token prediction but I think most folks are doing instruction tuning
Hi! Here is a example of instruction fine-tuning using samsum dataset where the input part has been masked out.
🚀 The feature, motivation and pitch
Is there a plan to add support for (FLAN style) instruction tuning?
Ideally, we should do summarization (SamSum dataset used in this repo) as instruction tuning as well. For example, we should not compute loss on the input part.
Alternatives
No response
Additional context
No response