Closed evanatyourservice closed 3 years ago
Hi, the “rectify” and “degenerate to sgd” is the same as in RAdam proposed in the paper “ on the variance of adaptive learning rate and beyond”. If rectify==False, then rectification is turned off and it does not matter whether degenerate is True or False. If rectify==True, I only tested with degenerate==True since it’s recommended in the RAdam paper (so typically I set degenerate==True, and play with rectify = True or False). I’m not sure about a general principle whether to turn on rectify or not. My experience is for experiments where Adam significantly outperforms sgd, such as SN-GAN, and the model is trained for a long time, rectification helps ( but I don’t have a good intuition for that)
Oh okay, thank you!
Hello and great work! I was wondering about the "degenerated_to_sgd" hyperparameter. Can you explain the background behind it and maybe provide a paper about it if there is one? Also, would you say the recommendations on when to use it are similar to rectify? If not, when do you think it should be used (beneficial all the time or only some of the time)?