Open artyomtugaryov opened 1 week ago
In the megatron/core/optimizer/__init__.py file the _get_param_groups function overides original lr_mult and setup groups of parameters uncorrectly. Thus, I propose the fix to kip the original value and setup parameters groups correctly.
megatron/core/optimizer/__init__.py
_get_param_groups
lr_mult
In the
megatron/core/optimizer/__init__.pyfile the_get_param_groupsfunction overides originallr_multand setup groups of parameters uncorrectly. Thus, I propose the fix to kip the original value and setup parameters groups correctly.