The function pretty_stats cannot read solver_specific flags

[DoctorDro]

Dear all, I am running benchmarks on some JuMP problems. Some of them are nonconvex and during the course of iterations I have a way of storing a nonconvex certificate. This however, is not standard and has to be stored in solver_specific flags using the commands:

stats = GenericExecutionStats(nlp, status=status, solution=x, objective=fx, dual_feas=norm(∇fx), iter=iter, elapsed_time=Δt)
set_solver_specific!(stats, Symbol("isConvex"), is_convex)
return stats

However, I get an error when I try the following

stats = bmark_solvers( solvers, problems, skipif=prob -> (!unconstrained(prob) || get_nvar(prob) > 1000 || get_nvar(prob) < 2))

combined_stats = DataFrame(
    name           = stats[:newton].name,
    nvars          = stats[:newton].nvar,
    convex         = stats[:newton].solver_specific[:isConvex],
    newton_iters   = stats[:newton].iter,
)

pretty_stats(combined_stats)

Any ideas how the DataFrame could read whether the problem is convex or not ?

[tmigot]

Hi @DoctorDro !

I tried to make a reproducible example and it seems to work! Note that you can access the solver specific fields directly stats[:newton].isConvex. Does that solve your problem?

using NLPModelsTest
using DataFrames, SolverCore, SolverBenchmark

function newton(nlp)
stats = GenericExecutionStats(nlp)
set_solver_specific!(stats, Symbol("isConvex"), :is_convex)
return stats
end

solvers = Dict(
    :newton => newton
)
problems = [NLPModelsTest.BROWNDEN()]
stats = bmark_solvers(solvers, problems)

combined_stats = DataFrame(
    name           = stats[:newton].name,
    nvars          = stats[:newton].nvar,
    convex         = stats[:newton].isConvex,
    newton_iters   = stats[:newton].iter,
)

pretty_stats(combined_stats)

[DoctorDro]

Hi @DoctorDro !

I tried to make a reproducible example and it seems to work! Note that you can access the solver specific fields directly stats[:newton].isConvex. Does that solve your problem?

using NLPModelsTest
using DataFrames, SolverCore, SolverBenchmark

function newton(nlp)
stats = GenericExecutionStats(nlp)
set_solver_specific!(stats, Symbol("isConvex"), :is_convex)
return stats
end

solvers = Dict(
    :newton => newton
)
problems = [NLPModelsTest.BROWNDEN()]
stats = bmark_solvers(solvers, problems)

combined_stats = DataFrame(
    name           = stats[:newton].name,
    nvars          = stats[:newton].nvar,
    convex         = stats[:newton].isConvex,
    newton_iters   = stats[:newton].iter,
)

pretty_stats(combined_stats)

You are amazing! Thanks! I was missing the : in :is_convex on top of other things. Have a hard time understanding symbols and other goodies Julia has introduced. Different logic that I have to get used to. Is there any version of pretty_stats which writes the output on a file or so?

[dpo]

Yes, you can pass an IOStream to pretty_stats: https://github.com/JuliaSmoothOptimizers/SolverBenchmark.jl/blob/24571ffbb92abd11652a62c9c10aadb2df40f166/src/formats.jl#L17 (e.g., an open file).

Example: https://github.com/JuliaSmoothOptimizers/SolverBenchmark.jl/blob/24571ffbb92abd11652a62c9c10aadb2df40f166/test/tables.jl#L29

[tmigot]

@DoctorDro I close this as I think we answer your questions. I opened a new issue to add the missing info in the documentation.