search

meta-llama / llama-recipes

Scripts for fine-tuning Meta Llama with composable FSDP & PEFT methods to cover single/multi-node GPUs. Supports default & custom datasets for applications such as summarization and Q&A. Supporting a number of candid inference solutions such as HF TGI, VLLM for local or cloud deployment. Demo apps to showcase Meta Llama for WhatsApp & Messenger.
11.98k stars 1.82k forks source link

BUG2 #416

Closed Phani1609 closed 1 month ago

Phani1609 commented 6 months ago

🚀 The feature, motivation and pitch

I propose adding a feature to improve the visualization of loss curves in LLaMA-Recipes. Currently, the loss curves may exhibit spikes or irregularities, making it challenging to interpret training progress accurately. By enhancing the loss curve visualization, users can better analyze model training dynamics, identify potential issues, and make informed decisions for model optimization.

Motivation: The motivation for this proposal stems from the need for clearer and more informative loss curve visualization in LLaMA-Recipes. Clearer visualization can aid researchers and practitioners in understanding model behavior, diagnosing training issues, and fine-tuning training strategies to improve model performance.

Pitch: The proposed feature involves implementing smoother and more informative loss curve visualization techniques in LLaMA-Recipes. This could include methods such as moving average smoothing, adaptive smoothing algorithms, or interactive visualization tools that allow users to explore and analyze loss curves dynamically.

Alternatives

Manual post-processing of loss curve data to achieve smoother visualization. Using external visualization libraries or tools to visualize loss curves outside of LLaMA-Recipes.

Additional context

No response

mreso commented 5 months ago

Thanks for the suggestions, we welcome contributions from the community.

HamidShojanazeri commented 4 months ago

@Phani1609 would it be something you are interested to add to W&B implementation here? https://github.com/meta-llama/llama-recipes/blob/main/src/llama_recipes/finetuning.py#L52

init27 commented 1 month ago

Closing but please re-open if you are interested in contributing to any feature, would be happy to help you @Phani1609. Thanks!