search

gonum / optimize

Packages for solving minimization problems [DEPRECATED]
66 stars 9 forks source link

Linesearchers should test function decrease to within a tolerance #128

Closed btracey closed 7 years ago

btracey commented 9 years ago

The linesearches we implement should not test for strict function decrease (fnew <= fmin), but instead test within a tolerance. This helps get closer to the true minimum (based on the gradient norm), as the function value converges faster than the gradient for a smooth problem near the optimum.

This was originally part of the line search, but was removed because of confusion with updating OptLoc. Now that the optimizer is strictly in charge of updating OptLoc, we can add this test back in.

It should be a tunable parameter. It should work properly with #127 .

vladimir-ch commented 9 years ago

Just a simple observation to support that we should do this: requiring strict decrease of fnew in Bisection breaks two tests for BFGS and LBFGS. Permitting <= allows to get to a point with smaller gradient norm, so if we permit even a tiny increase, we could get closer to the minimizer for some problems.

btracey commented 7 years ago

We tried this, and it turned out to be a bad idea. We replaced this with the use of MajorIteration for the optimizer to control when stats get updated.