Implement Optimizers

[bhavnicksm]

This issue is to keep track of all the optimizers to implement in the repository.

To Do

• [x] SGD (.SGD)

• [x] Momentum (.Momentum)

• [x] RMSProp (.RMSProp)

• [x] AdaGrad (.Adagrad)

• [x] Adam (.Adam)

• [x] AdaMax (.Adamax)

• [x] AdaDelta (.Adadelta)

• [x] #27

• [x] AMSGrad (AMSGrad)

• [x] RAdam (.Radam)

• [x] LION (.Lion)

• [x] #31

• [x] NAdam (.NAdam)

• [x] AdaBelief (.Adabelief)

• [ ] Adan (.Adan)

• [ ] AdaFactor(.Adafactor)

• [ ] AccSGD (.AccSGD)

• [ ] AdaBound (.Adabound)

• [ ] AdaMod (.AdaMod)

• [ ] Demon SGD/Adam (.DemonSGD & .DemonAdam)

• [ ] AggMo (.AggMo)

• [ ] QHM (.QHM)

• [ ] QHAdam (.QHAdam)

• [ ] LAMB (.Lamb)

• [ ] SWATS (.Swats)

• [ ] Shampoo (.Shampoo)

• [ ] SAM (.SAM)

• [ ] AdamP (.AdamP)

• [ ] Sophia (.Sophia)

• [ ] GaLore (.GaLore)

Resources

• Blog on writing custom optimisers by Daniel Mcneela
• Source code of Torch.Optim.SGD
• Source code of Torch.Optim.Adam
• Blog on difference of PyTorch Implementation and Normal SGD
• Ways to visualize a optimizer
• LabLab's implementation of the Adam and other Optimisers