search

hahnec / torchimize

numerical optimization in pytorch
https://hahnec.github.io/torchimize/
GNU General Public License v3.0
134 stars 7 forks source link

tip for possible perf boost on gauss-newton #1

Closed hjalmarlucius closed 2 years ago

hjalmarlucius commented 2 years ago

https://github.com/hahnec/torchimize/blob/e0dcfc6eb8a8c86133dbdf17f8fbd35633a25d16/torchimize/functions/gna_fun.py#L67

Replace line with h = -l * torch.linalg.lstsq(j, f, rcond=None, driver=None)[0] and skip the other matmuls and transposes.

hahnec commented 2 years ago

If I get you right, you are proposing the classical Gradient Descent method (first-order) whereas the "lsq_gna" function is the Gauss-Newton method using a second order term. I am thinking of adding more optimization algorithms over time such as Newton-Raphson. Thanks for the suggestion anyways.

hjalmarlucius commented 2 years ago

My proposal should give you the very same result as what you have already, just with two less matmuls. torch.linalg.lstsq solves using the pseudoinverse so you don't need to calculate both sides with J.T first. This doesn't work for the LM computation however since lstsq can't apply Tikonov regularization.

hahnec commented 2 years ago

Circling back on this, I just wanted to inform you that gradient descent is now featured in a single and parallelized fashion from v0.0.12.