search

meta-llama / llama-recipes

Scripts for fine-tuning Meta Llama3 with composable FSDP & PEFT methods to cover single/multi-node GPUs. Supports default & custom datasets for applications such as summarization and Q&A. Supporting a number of candid inference solutions such as HF TGI, VLLM for local or cloud deployment. Demo apps to showcase Meta Llama3 for WhatsApp & Messenger.
11.52k stars 1.63k forks source link

Instruction tuning support #102

Closed vivekmadan2 closed 2 weeks ago

vivekmadan2 commented 1 year ago

🚀 The feature, motivation and pitch

Is there a plan to add support for (FLAN style) instruction tuning?

Ideally, we should do summarization (SamSum dataset used in this repo) as instruction tuning as well. For example, we should not compute loss on the input part.

Alternatives

No response

Additional context

No response

findalexli commented 10 months ago

Any support on this? We found out that the training is doing just next token prediction but I think most folks are doing instruction tuning

wukaixingxp commented 3 months ago

Hi! Here is a example of instruction fine-tuning using samsum dataset where the input part has been masked out.