[x] Your code builds clean without any errors or warnings
[x] You are using approved terminology
[x] You have added unit tests
Motivation
Fix the bug with weight decay when set filter_bias_and_bn:
When filter_bias_and_bn is True: function init_group_params does not set value of weight decay for no_decay_params, in this case, mindspore will use weight_decay of optimizer (usually not 0.0).
When filter_bias_and_bn is False: mindspore will automatically filter BatchNorm params from weight_decay.
So the name of the argument is not the same as what it actually does. And we can never filter out the param of bias and norm layer from doing weight decay, as the name filter_bias_and_bn.
Due to this, we refactor it to weight_decay_filter:
"disable": No parameters to filter.
"auto": We do not apply weight decay filtering to any parameters. However, MindSpore currently automatically filters the parameters of Norm layer from weight decay.
"norm_and_bias": Filter the paramters of Norm layer and Bias from weight decay.
How do I migrate from an old configuration?
filter_bias_and_bn
weight_decay_filter
True
"disable"
False
"auto"
Keep in mind that filter_bias_and_bn doesn't do what its name suggests.
BTW, we also support get no_weight_decay list from model and layer_decay.
Test Plan
(How should this PR be tested? Do you require special setup to run the test or repro the fixed bug?)
Thank you for your contribution to the MindCV repo. Before submitting this PR, please make sure:
Motivation
Fix the bug with weight decay when set
filter_bias_and_bn:filter_bias_and_bnisTrue: functioninit_group_paramsdoes not set value of weight decay forno_decay_params, in this case, mindspore will useweight_decayof optimizer (usually not 0.0).filter_bias_and_bnisFalse: mindspore will automatically filter BatchNorm params from weight_decay.So the name of the argument is not the same as what it actually does. And we can never filter out the param of bias and norm layer from doing weight decay, as the name
filter_bias_and_bn.Due to this, we refactor it to
weight_decay_filter:"disable": No parameters to filter."auto": We do not apply weight decay filtering to any parameters. However, MindSpore currently automatically filters the parameters of Norm layer from weight decay."norm_and_bias": Filter the paramters of Norm layer and Bias from weight decay.True"disable"False"auto"BTW, we also support get no_weight_decay list from model and layer_decay.
Test Plan
(How should this PR be tested? Do you require special setup to run the test or repro the fixed bug?)
Related Issues and PRs
(Is this PR part of a group of changes? Link the other relevant PRs and Issues here. Use https://help.github.com/en/articles/closing-issues-using-keywords for help on GitHub syntax)