search

NVIDIA / NeMo

A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech)
https://docs.nvidia.com/nemo-framework/user-guide/latest/overview.html
Apache License 2.0
11.7k stars 2.44k forks source link

Missing Support for MultiStepLR in NeMo Framework #8770

Closed Qmsla1 closed 4 months ago

Qmsla1 commented 6 months ago

The NVIDIA NeMo Framework, a powerful generative AI framework, currently lacks support for the MultiStepLR learning rate scheduler, which is available in PyTorch’s torch.optim.lr_scheduler module. This scheduler is commonly used for adjusting learning rates at specific milestones during training.

Expected Behavior: NeMo should include an implementation or provide guidance on how to use MultiStepLR effectively within its training pipelines.

Steps to Reproduce:

Attempt to use MultiStepLR with NeMo conf yaml. Observe that it is not directly supported. Additional Context: Support for MultiStepLR would enhance the flexibility and adaptability of NeMo’s training process, especially for researchers and developers working on large language models, multimodal models, automatic speech recognition, and text-to-speech synthesis.

how to fix: add MultiStepLR to https://github.com/NVIDIA/NeMo/blob/main/nemo/core/config/schedulers.py

nithinraok commented 6 months ago

Please feel free to put a PR with this added lr. You can add it directly from pt scheduler similar to other schedulers as shown here: https://github.com/NVIDIA/NeMo/blob/6ca22351ea7dd231dfb94284d141de938cdb2822/nemo/core/optim/lr_scheduler.py#L963

github-actions[bot] commented 5 months ago

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

github-actions[bot] commented 4 months ago

This issue was closed because it has been inactive for 7 days since being marked as stale.