search

f-dangel / backpack

BackPACK - a backpropagation package built on top of PyTorch which efficiently computes quantities other than the gradient.
https://backpack.pt/
MIT License
549 stars 55 forks source link

[Feature Request] Levenberg Marquardt #293

Open mizu-bai opened 1 year ago

mizu-bai commented 1 year ago

Feature

Implementation of Levenberg Marquardt optimizer.

Motivation

In MATLAB, the feedforwardnet supports many optimizers, and some are not implemented in PyTorch, such as trainlm (Levenberg Marquardt). From tests from our gourp and others, Levenberg Marquardt performs much better and needs less epochs to coverage than first-order optimizers (Adam, SGD with Momentum) on regression model. Some issues in PyTorch issue list hope for a implementation of Levenverg Marquardt algorithm, but there are no more reponse. It will be helpful to implement the Levenberg Marquardt optimizer and I am willing to help test.

Addition

Here are some helpful information and codes.

f-dangel commented 1 year ago

Hi,

interesting suggestion. Do you have a specific use case where first-order methods don't work for example? BackPACK certainly provides the functionality to implement such an optimizer, and I would be happy to help out in case of issues.