Solvers are functions, problem lists are generators

[dpo]

This PR changes the benchmark tools so that

• solvers are functions, not symbols
• problem sets contain AbstractNLPModels but are generators so the models are only instantiated when needed. Thus run_mpb_problem() and run_ampl_problem() are no longer useful.

There is some overlap with #28.

[coveralls]

Coverage decreased (-1.5%) to 69.123% when pulling bff7fcd708302110dc9c03fe855797a5b3141841 on generators into e182fe5596ff7772973d8057291e7b6bc8f5d227 on master.

[coveralls]

Coverage decreased (-1.5%) to 69.123% when pulling bff7fcd708302110dc9c03fe855797a5b3141841 on generators into e182fe5596ff7772973d8057291e7b6bc8f5d227 on master.

[dpo]

There doesn't seem to be a supertype for generators, so it looks like I have to use Any.

[coveralls]

Coverage decreased (-0.5%) to 70.47% when pulling ab40083ac987dbdcd7a4f4a5b84aea055c7c0201 on generators into aa5ec2e94374d3042013d75029fc91fab59b2775 on master.

[coveralls]

Coverage decreased (-1.7%) to 69.338% when pulling f949e307629ff5acca6b3492bb63bc0af8f85efa on generators into aa5ec2e94374d3042013d75029fc91fab59b2775 on master.

[dpo]

Not sure what's going on here: https://travis-ci.org/JuliaSmoothOptimizers/Optimize.jl/jobs/169105504#L569

[vepiteski]

There remain many instances of Symbols, :trunk instead of trunk

Why not add a solve_problems_filtered that remove the skipped instances from the stats ?

[vepiteski]

In #28 I used stats, a dict of Function but I prefer the solution here, a dict of Symbol which may represent Symbol(function_name).

[dpo]

I first only focused on the two_solvers() benchmark example. Symbols should be removed now.

There's a new optional argument prune to solve_problems() which defaults to true and removes skipped problems from stats. That means the number of problems can only be determined after the benchmarks were run. That now happens in profile_solvers().

Finally, bmark_and_profile() returns both the stats and profile.

[dpo]

I guess this error with Julia 0.4 is due to generator expressions only being available in 0.5?

[abelsiqueira]

Looks like it. Maybe use array compreheension to pass?

[dpo]

Great idea.

[coveralls]

Coverage decreased (-2.6%) to 68.385% when pulling 0dd814c62fb88811dd0dbd6f9a6ab9b0931f10d9 on generators into aa5ec2e94374d3042013d75029fc91fab59b2775 on master.

[coveralls]

Coverage decreased (-2.6%) to 68.385% when pulling 56db1c3a485867461730315daa1c509c2853f1ac on generators into aa5ec2e94374d3042013d75029fc91fab59b2775 on master.

[coveralls]

Coverage decreased (-2.6%) to 68.385% when pulling 56db1c3a485867461730315daa1c509c2853f1ac on generators into aa5ec2e94374d3042013d75029fc91fab59b2775 on master.

[coveralls]

Coverage decreased (-2.6%) to 68.385% when pulling 56db1c3a485867461730315daa1c509c2853f1ac on generators into aa5ec2e94374d3042013d75029fc91fab59b2775 on master.

[coveralls]

Coverage decreased (-2.6%) to 68.385% when pulling 56db1c3a485867461730315daa1c509c2853f1ac on generators into aa5ec2e94374d3042013d75029fc91fab59b2775 on master.

[vepiteski]

How to modify this branch, which is already a PR? A PR within a PR?

For the moment, I have only the following 2 small corrections:

• in bmark_and_profile, use twice profile or profiles, but not profile and profiles
• in bmark_solvers, add at the beginning of the loop
  @printf("\n%-15s %s\n\n", " running ",string(solver))

[vepiteski]

More general concerns.

• Benchmarks stat f+g+hprod. Using appropriate automatic differentiation, the cost of f,g and hprod are comparable. What when comparing with a full hessian second order variant?
• stopping criteria should reflect this. Now, trunk and lbfgs do stop on #f alone tired = nlp.counters.neval_obj > max_f In my variants from vepiteski/ARCTR.jl, I rather use calls = [nlp.counters.neval_obj, nlp.counters.neval_grad, nlp.counters.neval_hess, nlp.counters.neval_hprod] tired = (iter >= itmax) | (sum(calls) > max_calls) but of course, a hess is much more costly than f, g or hprod. I am note sure to keep itmax.

[dpo]

I fixed "profiles vs. profile" and print the solver name. The discussion about stopping criteria should go to a general issue (it's not strictly related to this PR).

[vepiteski]

OK. Other small suggestions:

• print the name of skipped problems, and the mention "skipped".
• test benchmarking (would have trapped profile vs profiles).

Remark: when executing the Pkg.test, problem's names are prefixed by "OptimizationProblems". I would like to remove the prefix. No such prefix is displayed when I benchmark ARCTR and I don't know why.

[dpo]

I think printing the name of skipped problems will pollute the output quite a bit. In the near future, I think we should use logging in all packages. That will allow us to direct messages to a file or somewhere else so we only see essential information on the screen. Does that sound ok?

Running benchmark on Travis is tricky because it requires the AMPL models. I guess we could include just a few, but does that really belong in this package? Should the benchmark stuff be taken out of here?

[dpo]

julia> using OptimizationProblems

julia> p = dixmaane
dixmaane (generic function with 2 methods)

julia> string(p)
"OptimizationProblems.dixmaane"

[coveralls]

Coverage decreased (-2.5%) to 68.493% when pulling 0e67863051577fbe823d21f32b606fef4c70c6ef on generators into aa5ec2e94374d3042013d75029fc91fab59b2775 on master.

[coveralls]

Coverage decreased (-2.5%) to 68.493% when pulling 0e67863051577fbe823d21f32b606fef4c70c6ef on generators into aa5ec2e94374d3042013d75029fc91fab59b2775 on master.

[coveralls]

Coverage decreased (-2.5%) to 68.493% when pulling 0e67863051577fbe823d21f32b606fef4c70c6ef on generators into aa5ec2e94374d3042013d75029fc91fab59b2775 on master.

[coveralls]

Coverage decreased (-2.5%) to 68.493% when pulling 0e67863051577fbe823d21f32b606fef4c70c6ef on generators into aa5ec2e94374d3042013d75029fc91fab59b2775 on master.

[vepiteski]

I don't see what is different when I benchmark ARCTR variants, but I get: the prefix ARCTR for the solver, but no prefix for the problems. Again, this discussion probably does not belong to this PR, but this behaviour remains a mystery. The ARCTR benchmarks are performed using the function compare_solvers, which would be a nice addition to Optimize.

Example 1b: benchmark several solvers on a set of problems

function compare_solvers(solvers,probs) bmark_args = Dict{Symbol, Any}(:skipif => model -> model.meta.ncon > 0) profile_args = Dict{Symbol, Any}(:title => "f+g+hprod") stats, profiles = bmark_and_profile(solvers, (MathProgNLPModel(eval(p)(n), name=string(p)) for p in probs), bmark_args=bmark_args, profile_args=profile_args) return stats, profiles end

[dpo]

I added a few nl files so the benchmark functions in examples/benchmark.jl are easier to run, but it's difficult to run them on Travis because profiles are not available on Julia 0.4. If we decide to drop Julia 0.4, things will become simpler.

[coveralls]

Coverage decreased (-2.5%) to 68.493% when pulling 92832cba79bf42ec05279acf15b34743fd1722a0 on generators into aa5ec2e94374d3042013d75029fc91fab59b2775 on master.

[coveralls]

Coverage decreased (-2.5%) to 68.493% when pulling 92832cba79bf42ec05279acf15b34743fd1722a0 on generators into aa5ec2e94374d3042013d75029fc91fab59b2775 on master.

[coveralls]

Coverage decreased (-2.5%) to 68.493% when pulling 92832cba79bf42ec05279acf15b34743fd1722a0 on generators into aa5ec2e94374d3042013d75029fc91fab59b2775 on master.

[abelsiqueira]

Thanks