search

rfeinman / pytorch-minimize

Newton and Quasi-Newton optimization with PyTorch
https://pytorch-minimize.readthedocs.io
MIT License
292 stars 34 forks source link

Levenberg-Marquardt exact #7

Closed bhalazs closed 2 years ago

bhalazs commented 2 years ago

Hi,

I really like your project as I had been looking for (quasi-) second-order optimization methods in Pytorch for a while. I am a chemical engineering PhD student. My "specialization" is combining differential equations using the torchdiffeq library with embedded neural networks.

This is the first time that I am contributing to a GitHub project. I've decided to start with an "exact" Levenberg-Marquardt implementation where I basically just modified your "Newton-Exact" function to implement LM regularization/damping. This is not the "canonical" form of LM with nonlinear least squares that only uses the Jacobian, but it's quite convenient and robust for training smaller neural networks that are often sufficient in the realm of engineering, or for parameter fitting of differential equations. That being said, I would also like to implement LM properly in the lstsq module, possibly with bounds, if you are interested.

Thanks for your great work with this repo.

rfeinman commented 2 years ago

Hi @bhalazs - thanks for contributing! It looks like you closed the PR so I assume you are making some changes. I can take a look if/when you have a working draft.

bhalazs commented 2 years ago

Hi, yeah I changed my mind because some testing of my "exact" LM implementation revealed that the options for regularizing non-positive-definite Hessians in your "newton-exact" implementation are in fact much better than my standard Levenberg-Marquardt regularization. I'm planning to implement LM for the nonlinear least squares case instead, based on the MINPACK paper. I'll open a new request when that's done.