Closed btracey closed 7 years ago
Just a simple observation to support that we should do this: requiring strict decrease of fnew in Bisection breaks two tests for BFGS and LBFGS. Permitting <= allows to get to a point with smaller gradient norm, so if we permit even a tiny increase, we could get closer to the minimizer for some problems.
We tried this, and it turned out to be a bad idea. We replaced this with the use of MajorIteration for the optimizer to control when stats get updated.
The linesearches we implement should not test for strict function decrease (fnew <= fmin), but instead test within a tolerance. This helps get closer to the true minimum (based on the gradient norm), as the function value converges faster than the gradient for a smooth problem near the optimum.
This was originally part of the line search, but was removed because of confusion with updating OptLoc. Now that the optimizer is strictly in charge of updating OptLoc, we can add this test back in.
It should be a tunable parameter. It should work properly with #127 .