Confusing behavior of f_abstol with Fminbox

[fredcallaway]

I'm having a hard time understanding why having a higher (more lenient) tolerance results in more iterations when using Fminbox.

using Optim
f(x) = (1.0 - x[1])^2 + 100.0 * (x[2] - x[1]^2)^2
tols = @. 10. ^ - (1:12)  # .1, .01 ...
map(tols) do tol
    # opt = optimize(f, zeros(2), LBFGS(), Optim.Options(f_abstol=tol))
    opt = optimize(f, -2ones(2), 2ones(2), zeros(2),  Fminbox(LBFGS()), Optim.Options(f_abstol=tol))
    opt.iterations
end |> println
# prints: [11, 6, 6, 4, 4, 4, 3, 3, 3, 3, 3, 3]

Note that if I switch to unbounded optimization (switch the comment) I get the expected behavior of increasing iterations: [3, 18, 21, 21, 23, 23, 23, 24, 24, 24, 24, 24]

Is this expected behavior?

More pragmatically, I have a bounded optimization problem where I only need three digits of precision and I can see that the optimizer is spending the vast majority of iterations within this range of the final result, no matter what I set f_abstol to.

[pkofod]

Because you are setting the inner tolerance while maintaining the outer tolerance of the barrier method. However, it appears that I never added outer_f_abstol checks anywhere.

Remember though, that setting f_abstol only controls the change in two consecutive evaluations. It's just telling you that there's little progress, not necessarily that you're close to the optimum.

[fredcallaway]

Indeed, I tried Optim.Options(f_abstol=tol, outer_f_abstol=tol) and this gave the same result. So it sounds like it is currently not possible to adjust the tolerance for box constrained optimization. Is that correct? Is this something that can be easily fixed?

[pkofod]

Yes, this is quite easy to fix. I will try to prioritize it. But please understand that f_abstol does not guarantee that you're in some proximity of a solution. It's just a restriction on the changes if f between iterations. Did you consider outer_g_abstol instead?

[fredcallaway]

To be honest, I don't know enough theory to understand how these practically differ. It seems to me that the change in f value should be directly related to the gradient. In particular, the problem of nearly flat regions that aren't near the global minimum applies just as much to g_abstol as to f_abstol, doesn't it?

In any case, I find that outer_g_abstol has no effect, at least for the Rosenbrock function. However, g_abstol does have the desired effect. I'm using the syntax Optim.Options(g_abstol=tol, outer_g_abstol=tol)

[pkofod]

I'm having a hard time understanding why having a higher (more lenient) tolerance results in more iterations when using Fminbox.

So I'm going to close the fact that these weren't passed to the convergence checker, but in general you should be aware that if you make the inner optimization more lenient you can expect to have more outer iterations and vice versa, but it's not always the case. There's a balance between following the central path more or less exactly.

[fredcallaway]

I'm leaving this here rather than making an issue because I think it might be essentially the same thing (and maybe has already been fixed on master)

It seems to me that f_calls_limit is ignored as well.

julia> f(x) =  (x[1] - 0.1) ^ 4 + (sum(x) - 0.5) ^ 2
julia> optimize(f, zeros(2), ones(2), rand(2), Fminbox(LBFGS()), Optim.Options(f_calls_limit = 10); autodiff=:forward)
...
 * Work counters                                                                                                                     
    Seconds run:   0  (vs limit Inf)                                                                                                 
    Iterations:    646                                                                                                               
    f(x) calls:    1048787                                                                                                           
    ∇f(x) calls:   1048787