Closed 21kc-caracol closed 2 months ago
It’s the second option. If you only have a limited number of trials, you can focus on just tuning the learning rate, but if you have more compute/time, you can optimize beta1 and LR in conjunction.
If you change beta1 (or beta2 etc), you’ll need to retune LR.
Please clarify if For a budget of 10-25:
I understood that its option 1. First tune for best learning rate, then fix that value, and start tuning beta1.