Scripts for fine-tuning Meta Llama3 with composable FSDP & PEFT methods to cover single/multi-node GPUs. Supports default & custom datasets for applications such as summarization and Q&A. Supporting a number of candid inference solutions such as HF TGI, VLLM for local or cloud deployment. Demo apps to showcase Meta Llama3 for WhatsApp & Messenger.
11.52k
stars
1.63k
forks
source link
Masking loss from labels associated with user turns #209
In
Concatenator.__call__()
https://github.com/facebookresearch/llama-recipes/blob/main/src/llama_recipes/datasets/utils.py#L38 labels are created by simply copying the input_ids. Shouldn't the loss from the labels associated with the user turns be masked too? See for example https://github.com/bigcode-project/starcoder/blob/main/chat/dialogues.py#L232 and https://huggingface.co/blog/starchat-alpha#masking-user-labels.