What is a solution, how are solvers compared

[odow]

I don't think it is sufficient just to provide the models. We also need some concept of what a solution is, and how two runs can be compared.

Perhaps the simplest is the in-sample lower bound (if minimizing), and the time-weighted integral of that over a fixed time limit for each solver?

We went for something far more explicit: https://odow.github.io/StochOptFormat/#problems-policies-and-algorithms. It's an out-of-sample evaluation of a fixed set of scenarios.

[bonnkleiford]

Ahh, yes! In fact, we will be providing solution reports which would include (1) result summary in csv (provided by the three solvers), (2) first stage solution (in csv), and (3) convergence bounds (in png).

Below is an example of a compact one I use for presentations:

But on separate files and folders, of course!

[odow]

Some things to think about:

• What is a solution to a model in MSPLib?
• How do you choose how many samples to use when simulating the reward?
• Is the simulation in-sample or out-of-sample?
• Can we run two solvers with common random numbers for the simulations, so that the variance between the solvers is decreased?
• Is the lower bound entering the CI of the simulated cost a useful stopping criteria?
• If two solvers have found similar estimates for the CI of simulated cost, have they found similar policies?
• What if the first-stage does not have a unique solution?
• How do you measure and compare risk-averse policies?

[odow]

I think your Bounds picture needs to plot the full confidence interval as band. Otherwise it's a bit misleading because Instance 2 crosses over.