only_fgh! doesn't work if algorithm does not need Hessian

[cossio]

f(x) = (1.0 - x[1])^2 + 100.0 * (x[2] - x[1]^2)^2
function g!(G, x)
  G[1] = -2.0 * (1.0 - x[1]) - 400.0 * (x[2] - x[1]^2) * x[1]
  G[2] = 200.0 * (x[2] - x[1]^2)
end
function h!(H, x)
  H[1, 1] = 2.0 - 400.0 * x[2] + 1200.0 * x[1]^2
  H[1, 2] = -400.0 * x[1]
  H[2, 1] = -400.0 * x[1]
  H[2, 2] = 200.0
end
function fg!(F,G,x)
  G == nothing || g!(G,x)
  F == nothing || return f(x)
  nothing
end
function fgh!(F,G,H,x)
  G == nothing || g!(G,x)
  H == nothing || h!(H,x)
  F == nothing || return f(x)
  nothing
end

I can use Optim.only_fg! with a gradient-free method:

import Optim
Optim.optimize(Optim.only_fg!(fg!), [0., 0.], Optim.NelderMead()) 
#= works fine even if NelderMead does not need gradient =#

This can be convenient, in the sense that you only have to write fg! and then your code works for all gradient free and gradient required algorithms, which can be useful e.g. for benchmarking purposes. As stated in the docs, the gradient free algorithm can call fg! with G === nothing to request the function value only and not the gradient.

However this behavior is broken for higher derivatives. Here NelderMead() does not need gradient nor hessian, and only_fgh! fails:

Optim.optimize(Optim.only_fgh!(fgh!), [0., 0.], Optim.NelderMead())

ERROR: MethodError: objects of type Nothing are not callable
Stacktrace:
 [1] (::getfield(NLSolversBase, Symbol("##50#51")){NLSolversBase.InplaceObjective{Nothing,Nothing,typeof(fgh!)},Float64})(::Array{Float64,1}) at /opt/julia-depot/packages/NLSolversBase/KG9Ie/src/objective_types/incomplete.jl:35
 [2] value!!(::NonDifferentiable{Float64,Array{Float64,1}}, ::Array{Float64,1}) at /opt/julia-depot/packages/NLSolversBase/KG9Ie/src/interface.jl:9
 [3] initial_state(::NelderMead{Optim.AffineSimplexer,Optim.AdaptiveParameters}, ::Optim.Options{Float64,Nothing}, ::NonDifferentiable{Float64,Array{Float64,1}}, ::Array{Float64,1}) at /opt/julia-depot/packages/Optim/Agd3B/src/multivariate/solvers/zeroth_order/nelder_mead.jl:158
 [4] optimize at /opt/julia-depot/packages/Optim/Agd3B/src/multivariate/optimize/optimize.jl:33 [inlined]
 [5] #optimize#87(::Bool, ::Symbol, ::Function, ::NLSolversBase.InplaceObjective{Nothing,Nothing,typeof(fgh!)}, ::Array{Float64,1}, ::NelderMead{Optim.AffineSimplexer,Optim.AdaptiveParameters}, ::Optim.Options{Float64,Nothing}) at /opt/julia-depot/packages/Optim/Agd3B/src/multivariate/optimize/interface.jl:116
 [6] optimize(::NLSolversBase.InplaceObjective{Nothing,Nothing,typeof(fgh!)}, ::Array{Float64,1}, ::NelderMead{Optim.AffineSimplexer,Optim.AdaptiveParameters}, ::Optim.Options{Float64,Nothing}) at /opt/julia-depot/packages/Optim/Agd3B/src/multivariate/optimize/interface.jl:115 (repeats 2 times)
 [7] top-level scope at none:0

Instead I'd expect NelderMead to call fgh! with both G===nothing and H===nothing, to request the function value only but not the gradient nor the Hessian.

Similarly, LBFGS() needs the gradient but not the Hessian, and again only_fgh! fails:

Optim.optimize(Optim.only_fgh!(fgh!), [0., 0.], Optim.LBFGS())

ERROR: MethodError: objects of type Nothing are not callable
Stacktrace:
 [1] (::getfield(NLSolversBase, Symbol("##56#57")){NLSolversBase.InplaceObjective{Nothing,Nothing,typeof(fgh!)},Float64})(::Array{Float64,1}, ::Array{Float64,1}) at /opt/julia-depot/packages/NLSolversBase/KG9Ie/src/objective_types/incomplete.jl:38
 [2] value_gradient!!(::OnceDifferentiable{Float64,Array{Float64,1},Array{Float64,1}}, ::Array{Float64,1}) at /opt/julia-depot/packages/NLSolversBase/KG9Ie/src/interface.jl:82
 [3] initial_state(::LBFGS{Nothing,LineSearches.InitialStatic{Float64},LineSearches.HagerZhang{Float64,Base.RefValue{Bool}},getfield(Optim, Symbol("##22#24"))}, ::Optim.Options{Float64,Nothing}, ::OnceDifferentiable{Float64,Array{Float64,1},Array{Float64,1}}, ::Array{Float64,1}) at /opt/julia-depot/packages/Optim/Agd3B/src/multivariate/solvers/first_order/l_bfgs.jl:158
 [4] optimize(::OnceDifferentiable{Float64,Array{Float64,1},Array{Float64,1}}, ::Array{Float64,1}, ::LBFGS{Nothing,LineSearches.InitialStatic{Float64},LineSearches.HagerZhang{Float64,Base.RefValue{Bool}},getfield(Optim, Symbol("##22#24"))}, ::Optim.Options{Float64,Nothing}) at /opt/julia-depot/packages/Optim/Agd3B/src/multivariate/optimize/optimize.jl:33
 [5] #optimize#87 at /opt/julia-depot/packages/Optim/Agd3B/src/multivariate/optimize/interface.jl:116 [inlined]
 [6] optimize(::NLSolversBase.InplaceObjective{Nothing,Nothing,typeof(fgh!)}, ::Array{Float64,1}, ::LBFGS{Nothing,LineSearches.InitialStatic{Float64},LineSearches.HagerZhang{Float64,Base.RefValue{Bool}},getfield(Optim, Symbol("##22#24"))}, ::Optim.Options{Float64,Nothing}) at /opt/julia-depot/packages/Optim/Agd3B/src/multivariate/optimize/interface.jl:115 (repeats 2 times)
 [7] top-level scope at none:0

In this case I'd expect LBFGS to call fgh! with H===nothing, to request the function value and the gradient but not the Hessian.

[pkofod]

Funny should should open this, as another user has also just made me aware of this. I'll have to think about it. But it'll be fixed.

[Nosferican]

Just to clarify, would this allow for a TwiceDifferentiable(f, g!, gf! = fg!) like syntax to allow for autodiff for the Hessian and providing an efficient fg! method?

[pkofod]

I think that should be the job of the constructor of a twicedifferentiable that takes in a oncedifferentiable. I have a new version of optim in the works and this part will be changed quite a bit, so I’m sort of only patching here.

[Nosferican]

Got it. Looking forward to the new design. 👍

[pkofod]

Got it. Looking forward to the new design. +1

You can be one of the lucky beta testers... I'll provide you with many internet points! (though they'll be myspace add credits...)

[longemen3000]

this seems like a problem in NLSolversBase

[pkofod]

Sorry everyone! Kept slipping out of my mind. Should work now.

[tyleransom]

Thanks so much! I'll give it a whirl soon.