Closed skeydan closed 3 years ago
Is there a specific reason we have not implemented strong_wolfe? If no, I can create a PR.
No, I think I just wanted to have optim_lbfgs
as fast as possible.
Is it a lot of effort to make our optimizer class extendable by other packages?
That should be fast, I can do that in the next few days.
Fixed in #517
I was comparing R and Python behavior on an optimization problem (an outcome of this is https://github.com/mlverse/torch/pull/515/files).
But the swallowed error aside, this problem really impressively demonstrates the importance of line search in some cases - compare (iteration 1 in both cases, for R, loss will be Inf thereafter):
R:
Python:
Is there a specific reason we have not implemented
strong_wolfe
? If no, I can create a PR.Additionally, I think it would also be great to port [parts of] https://github.com/hjmshi/PyTorch-LBFGS, which - among others - has a few additional line search functions that work with larger datasets. I think we want to have an extra package (like
torchoptimizers
), right? Is it a lot of effort to make our optimizer class extendable by other packages?