Add support for SAM (Sharpness Awareness Minimization) based optimization

[fcakyon]

Add support for SAM based optimization

Motivation In Deep Learning we use optimization algorithms such as SGD/Adam to achieve convergence in our model, which leads to finding the global minima, i.e a point where the loss of the training dataset is low. But several kinds of research such as Zhang et al have shown, many networks can easily memorize the training data and have the capacity to readily overfit, To prevent this problem and add more generalization, Researchers at Google have published a new paper called Sharpness Awareness Minimization which provides State of the Art results on CIFAR10 and other datasets.

Related resources Pytorch based implementation: https://github.com/davda54/sam

[xvjiarui]

Hi @fcakyon We don't have plan for it for now. But PRs are welcome.

[guillaumefrd]

I would also be interested in the addition of this feature.