ordabayevy
commented
1 year ago Some useful links about lr schedulers:
- https://www.kaggle.com/code/residentmario/cosine-annealed-warm-restart-learning-schedulers
- https://www.kaggle.com/code/residentmario/one-cycle-learning-rate-schedulers/notebook
- https://sgugger.github.io/the-1cycle-policy.html
- https://timm.fast.ai/SGDR
- SGDR: STOCHASTIC GRADIENT DESCENT WITH WARM RESTARTS
- A DISCIPLINED APPROACH TO NEURAL NETWORK HYPER-PARAMETERS
- Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates
(unrelated to this PR -- but I have a feeling that scheduling the LR e.g. w/ the usual cosine annealing might do wonders with PPCA. why? because it's a simple model w/ a convex loss and after the initial convergence to a rough solution, annealing can let the "ball" roll down to the global minimum. It'd be great to try this out in your experiments!).
_Originally posted by @mbabadi in https://github.com/cellarium-ai/scvi-distributed/pull/6#discussion_r1099041575_