Solver name displayed after Info: ?

[DoctorDro]

Dear all,

is there any possibility to enable the printing of the solver being tested after the Info: ? Because if you are testing several of them and something is wrong or you see some exception, you do not know which solver concerns.

Info: as a string is useless by the way, since it is obvious that what is printed is information. Possibly Info: should be replaced by the name of the solver being tested Newton: or LBFGS:?

Any ideas?

[tmigot]

Hi! That's a good suggestion.

If I keep the same example as in #149

using NLPModelsTest
using DataFrames, SolverCore, SolverBenchmark

function newton(nlp)
stats = GenericExecutionStats(nlp)
set_solver_specific!(stats, Symbol("isConvex"), :is_convex)
return stats
end
lbfgs(nlp) = newton(nlp)

solvers = Dict(
    :Newton => newton,
    :LBFGS => lbfgs
)
problems = [NLPModelsTest.BROWNDEN()]
stats = bmark_solvers(solvers, problems)

Modifying this line https://github.com/JuliaSmoothOptimizers/SolverBenchmark.jl/blob/24571ffbb92abd11652a62c9c10aadb2df40f166/src/bmark_solvers.jl#L21 for @info "Running solver $(name) with function $(solver)" for instance would return

julia> stats = bmark_solvers(solvers, problems)
[ Info: Running solver LBFGS with function lbfgs
[ Info:            Name    nvar    ncon           status      Time      f(x)      Dual    Primal
[ Info: BROWNDEN_manual       4       0          unknown       Inf       Inf       Inf   0.0e+00
[ Info: Running solver Newton with function newton
[ Info:            Name    nvar    ncon           status      Time      f(x)      Dual    Primal
[ Info: BROWNDEN_manual       4       0          unknown       Inf       Inf       Inf   0.0e+00

Would that solve your issue?

[DoctorDro]

Hi @tmigot, thanks for your answer. What you suggested would definitely providing some missing info, which would also be lost when running all the CUTEst problems. What about removing the Info completely since what you print is information, or informative, and Info is therefore a logger convention, and replacing it with something like:

[ ------>  Running solver **LBFGS** with function _lbfgs_
[ LBFGS :            Name    nvar    ncon           status      Time      f(x)      Dual    Primal
[ LBFGS : BROWNDEN_manual       4       0          unknown       Inf       Inf       Inf   0.0e+00

[ ------>  Running solver **Newton** with function _newton_
[ Newton:            Name    nvar    ncon           status      Time      f(x)      Dual    Primal
[ Newton: BROWNDEN_manual       4       0          unknown       Inf       Inf       Inf   0.0e+00

I used bold for the solver an italics for the function but they are not formatted properly in the code

[tmigot]

The Info actually comes from our usage of the Julia macro @info https://docs.julialang.org/en/v1/stdlib/Logging/. It should be possible to define new level of logs to avoid the Info message though. If you have code suggestions, they are very welcome in a pull request.

[DoctorDro]

Thank you for the info ! What would you think about @info [$(name)] then? That would produce output like the following:

[ Info: Running solver LBFGS with function lbfgs [ Info: [LBFGS] Name nvar ncon status Time f(x) Dual Primal [ Info: [LBFGS] BROWNDEN_manual 4 0 unknown Inf Inf Inf 0.0e+00

[tmigot]

I like your suggestion. Do you want to open a PR for this?

[DoctorDro]

Happy you like it. It is hard to find people who are flexible nowadays. Do we need however, a new PR for this? Isn't it covered by the title of the present PR?

[tmigot]

This is an issue. a Pull Request would be an actual change to the code :)

[DoctorDro]

Sorry, I was confused I thought you meant another issue. I am not familiar with all these abbreviations. There are so many. So, how is a Pull Request different, what am I supposed to do? I am not too familiar with Julia packages.

[tmigot]

No problem, feel free to let me know if that sounds too much work. A pull request is a proposal to merge a set of changes from one branch into another.

As you may have noticed the code of this package is available here https://github.com/JuliaSmoothOptimizers/SolverBenchmark.jl , and to modify it, you need to open a pull request that will then be reviewed and validated before added to the code.

I think the Github procedure is described here https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&ved=2ahUKEwjYyObentOEAxVhGVkFHSnAAWcQFnoECCYQAQ&url=https%3A%2F%2Fdocs.github.com%2Farticles%2Fcreating-a-pull-request&usg=AOvVaw0OafNHvMFfUq56q9rPOrPT&opi=89978449 . If you decide to go for it, I can pinpoint the places where the code needs to be changed.

[DoctorDro]

Perfect let me know. I was also thinking of adding the number of iterations in a separate column if you also agree. They are important just before the time (time per iteration) * iterations is the total time. You can get a measure how cheap the iterations are.

[tmigot]

The line to be changed for the header: https://github.com/JuliaSmoothOptimizers/SolverBenchmark.jl/blob/bbec05989ff2a1e06e7e5d9f0db0d3aebe6b9310/src/bmark_solvers.jl#L21 The line that prints after each problem: https://github.com/JuliaSmoothOptimizers/SolverBenchmark.jl/blob/bbec05989ff2a1e06e7e5d9f0db0d3aebe6b9310/src/run_solver.jl#L127

I am too fond of the iteration count, because the definition of an iteration highly depend from one solver to another. It still is an information you can get from the result of the bmark_solvers.

[dpo]

Why isn’t it sufficient to enable debug logging with the code as it is to see the solver and problem name? Something like

using Logging

logger = ConsoleLogger(stdout, Debug)
stats = with_logger(logger)
  bmark_solvers(solvers, problems)
end

[tmigot]

If we use the debug mode it gives:

julia> stats = with_logger(logger) do                                                                         
       bmark_solvers(solvers, problems)                                                                       
       end
┌ Debug: running
│   name = :LBFGS
│   solver = lbfgs (generic function with 1 method)
└ @ SolverBenchmark C:\Users\tangi\.julia\dev\SolverBenchmark.jl\src\bmark_solvers.jl:21
[ Info:            Name    nvar    ncon           status      Time      f(x)      Dual    Primal
[ Info: BROWNDEN_manual       4       0          unknown       Inf       Inf       Inf   0.0e+00
┌ Debug: running
│   name = :Newton
│   solver = newton (generic function with 1 method)
└ @ SolverBenchmark C:\Users\tangi\.julia\dev\SolverBenchmark.jl\src\bmark_solvers.jl:21
[ Info:            Name    nvar    ncon           status      Time      f(x)      Dual    Primal
[ Info: BROWNDEN_manual       4       0          unknown       Inf       Inf       Inf   0.0e+00

which is also good. In a sense that it is slightly weird to have the @info printed without the @debug header. I think either we use @debug for all or @info.

Then, I understand @DoctorDro idea to have the name of the solver printed somewhere in case you solve a ton of problems.

[DoctorDro]

Thank you @tmigot . Just to explain to @dpo, that CUTEst consists of 1000s of problems. You would like to read the results one after the other. Debug modes makes them unreadable. You cannot show this output to anyone without throwing it back to you asking for a LaTeX table or an Excel-style document. We want to maintain beautiful output with a bit more info that is missing to complete the picture. Finally, in other words, you make sure your latest edits to your solver are incorporated, stopping criteria are updated, limit conditions are handled properly, and then you press the enter and leave your office to get a coffee and go for a walk. When you come back at your office the Info: output should answer completely the question: "What happened while I was out?".

[DoctorDro]

Dear @tmigot, I tried I cannot understand how to do the pull request. The information is not crystal clear. I click create a new pull request and then it is unclear what to do.

In my understanding a pull request proceeds the "cloning" of the repository. For instance I create a pull request, you check who am I and why do I want to pull, what am I planning to implement, and if you agree you approve my request and you allow me to clone. It seems it is the other way round, you need to fork, and it is not clear what to do with the fork, and something that could be described in one page, it is split across several pages and the whole process needs to be deciphered. Isn't it easier for you to change a few code lines? I would still go through it, if you could list me the steps.

[tmigot]

Hey @DoctorDro ! Sorry, I know it is not trivial. @dpo opened a PR to at least add the name of the solver in the info section. I let the other part in a new issue #154 for someone to grab.

Regarding Github, the process is more as follows: -> create a fork of the repository you plan to contribute to. -> make a new branch in this fork. -> add your new code (and test that it works) -> open a Pull Request from this branch of your fork to the main repository. I hope it clarifies it.

[DoctorDro]

Hey @DoctorDro ! Sorry, I know it is not trivial. @dpo opened a PR to at least add the name of the solver in the info section. I let the other part in a new issue #154 for someone to grab.

Regarding Github, the process is more as follows: -> create a fork of the repository you plan to contribute to. -> make a new branch in this fork. -> add your new code (and test that it works) -> open a Pull Request from this branch of your fork to the main repository. I hope it clarifies it.

Would you please tell me one reason why they do not describe the process as simply as you just described it to me giving also a simple example like a small tutorial in steps: step 1, step 2, step 3, ... ?