Closed odow closed 8 months ago
I am curious about how much this impacts the results on NLPModels. It can live in the variants for testing, but we should defer to the developers of NLPModels, if they want this to be the default implementation or not.
on the 118 case, it went from
Dict{String, Any} with 10 entries:
"cost" => 97213.6
"variables" => 1088
"constraints" => 1539
"case" => "data/pglib_opf_case118_ieee.m"
"time_total" => 3.76066
"time_build" => 1.11021
"solution" => Dict("p_102_66_65"=>-1.66211, "p_132_84_85"=>-0…
"time_solve" => 2.60632
"time_data" => 0.0441308
"feasible" => true
to
Dict{String, Any} with 10 entries:
"cost" => 97213.6
"variables" => 1088
"constraints" => 1539
"case" => "data/pglib_opf_case118_ieee.m"
"time_total" => 2.30815
"time_build" => 1.22802
"solution" => Dict("p_102_66_65"=>-1.66211, "p_132_84_85"=>-0…
"time_solve" => 1.03217
"time_data" => 0.0479541
"feasible" => true
so the solve is much faster
The 793 case went from
Summary
case........: data/pglib_opf_case793_goc.m
variables...: 5432
constraints.: 7978
feasible....: true
cost........: 260198
total time..: 46.25211310386658
data time.: 0.1509079933166504
build time: 27.710966110229492
solve time: 18.390239000320435
to
Summary
case........: data/pglib_opf_case793_goc.m
variables...: 5432
constraints.: 7978
feasible....: true
cost........: 260198
total time..: 39.35335683822632
data time.: 0.1566319465637207
build time: 29.251823902130127
solve time: 9.94490098953247
So again, the solve time is 2X faster. But still only a small reduction in total runtime from 46 seconds to 39 seconds.
@tmigot @amontoison thoughts?
Hi @ccoffrin @odow ! Thank you for maintaining this and the feedback. I noticed that ADNLPModels had new failures with the last update, but I didn't have the time to investigate.
In general, I think type-stable functions behave better in Julia, so it helps the auto-diff. Now, the slower build time is maybe because the function tree becomes more complicated !? Although this is just a wild guess.
Overall, I am fine with both solutions, it depends more on the benchmark philosophy. Either we compare one formulation for all modeling tools, or we also compete on what is the best formulation for each modeling tool. What I don't like about the latter is that it opens the door to people always saying that the benchmark will be biais because you didn't find the best formulation, but it will still be informative.
@tmigot thank you for following up! I have completed a detailed study of the two models for your consideration. The key decisions I would like you to provide are,
While you consider these choices, let me note that, my original inception of "rosetta-opf" was to be an educational tool for transfer learning in the Julia optimization ecosystem (like rosetta code is for programming languages). It should help folks understand how to model across the different frameworks. Over time it has become well known as a performance benchmark, however I still say this is not the primary objective of the project. For example, the Symbolic AD variant of the JuMP model is more performant on AC-OPF than the default one. However, I have not made it the default implementation here because it is not the best default AD choice for NLP modeling in JuMP.
With that said, here are the two versions of NLPModels we now have,
NLPModels - this is the current implementation NLPModels-CS - type-stable implementation using ConcreteStructs
Based on the table below the NLPModels-CS version is consistently faster (around 10%-25% in the large size limit).
Case | Vars | Cons | NLPModels | NLPModels-CS | NLPM/NLPM-CS |
---|---|---|---|---|---|
case3_lmbd | 24 | 28 | 5.10e-02 | 2.89e-02 | 1.77 |
case5_pjm | 44 | 53 | 1.58e-01 | 1.15e-01 | 1.37 |
case14_ieee | 118 | 169 | 2.73e-01 | 1.99e-01 | 1.37 |
case24_ieee_rts | 266 | 315 | 7.72e-01 | 5.81e-01 | 1.33 |
case30_ieee | 236 | 348 | 7.03e-01 | 5.00e-01 | 1.40 |
case30_as | 236 | 348 | 6.32e-01 | 5.69e-01 | 1.11 |
case39_epri | 282 | 401 | 6.59e-01 | 4.05e-01 | 1.63 |
case57_ieee | 448 | 675 | 1.05e+00 | 1.09e+00 | 0.97 |
case60_c | 518 | 737 | 1.84e+00 | 9.22e-01 | 2.00 |
case73_ieee_rts | 824 | 987 | 2.56e+00 | 1.53e+00 | 1.68 |
case89_pegase | 1042 | 1649 | 6.96e+00 | 3.84e+00 | 1.81 |
case118_ieee | 1088 | 1539 | 5.36e+00 | 2.45e+00 | 2.19 |
case162_ieee_dtc | 1484 | 2313 | 6.69e+00 | 4.50e+00 | 1.49 |
case179_goc | 1468 | 2200 | 9.79e+00 | 4.63e+00 | 2.11 |
case197_snem | 1608 | 2397 | 6.88e+00 | 4.47e+00 | 1.54 |
case200_activ | 1456 | 2116 | 5.47e+00 | 3.57e+00 | 1.53 |
case240_pserc | 2558 | 3617 | 5.16e+01 | 1.82e+01 | 2.83 |
case300_ieee | 2382 | 3478 | 1.22e+01 | 7.64e+00 | 1.59 |
case500_goc | 4254 | 6097 | 3.01e+01 | 1.84e+01 | 1.64 |
case588_sdet | 4110 | 5979 | 2.31e+01 | 1.56e+01 | 1.48 |
case793_goc | 5432 | 7978 | 3.80e+01 | 2.64e+01 | 1.44 |
case1354_pegase | 11192 | 16646 | 1.61e+02 | 1.12e+02 | 1.44 |
case1803_snem | 15246 | 23172 | 3.37e+02 | 2.03e+02 | 1.66 |
case1888_rte | 14480 | 21494 | 5.32e+02 | N.D. | N.D. |
case1951_rte | 15018 | 22075 | 3.30e+02 | 2.26e+02 | 1.46 |
case2000_goc | 19008 | 29432 | 4.05e+02 | 3.30e+02 | 1.23 |
case2312_goc | 17128 | 25716 | 3.14e+02 | 2.16e+02 | 1.45 |
case2383wp_k | 17004 | 25039 | 2.76e+02 | 1.95e+02 | 1.42 |
case2736sp_k | 19088 | 28356 | 3.04e+02 | 2.48e+02 | 1.23 |
case2737sop_k | 18988 | 28358 | 2.89e+02 | 2.40e+02 | 1.20 |
case2742_goc | 24540 | 38196 | 1.12e+03 | 7.05e+02 | 1.58 |
case2746wp_k | 19520 | 28446 | 3.13e+02 | 2.45e+02 | 1.28 |
case2746wop_k | 19582 | 28642 | N.D. | 2.56e+02 | N.D. |
case2848_rte | 21822 | 32129 | 6.41e+02 | 4.26e+02 | 1.51 |
case2853_sdet | 23028 | 33154 | 8.73e+02 | 5.33e+02 | 1.64 |
case2868_rte | 22090 | 32393 | 7.12e+02 | 4.80e+02 | 1.48 |
case2869_pegase | 25086 | 37813 | 8.32e+02 | 6.72e+02 | 1.24 |
case3012wp_k | 21082 | 31029 | 4.36e+02 | 3.44e+02 | 1.27 |
case3022_goc | 23238 | 34990 | 7.99e+02 | 5.53e+02 | 1.45 |
case3120sp_k | 21608 | 32092 | 4.61e+02 | 3.69e+02 | 1.25 |
case3375wp_k | 24350 | 35876 | 6.49e+02 | 5.33e+02 | 1.22 |
case3970_goc | 35270 | 54428 | 1.97e+03 | 1.53e+03 | 1.28 |
case4020_goc | 36696 | 56957 | 2.07e+03 | 1.62e+03 | 1.28 |
case4601_goc | 38814 | 59596 | 2.30e+03 | 1.95e+03 | 1.18 |
case4619_goc | 42532 | 66289 | 2.47e+03 | 2.30e+03 | 1.07 |
case4661_sdet | 34758 | 51302 | 1.51e+03 | 1.24e+03 | 1.21 |
case4837_goc | 41398 | 64030 | 2.24e+03 | 2.05e+03 | 1.09 |
case4917_goc | 37872 | 56917 | 2.12e+03 | 1.68e+03 | 1.26 |
case5658_epigrids | 48552 | 74821 | 3.41e+03 | 3.04e+03 | 1.12 |
case6468_rte | 49734 | 75937 | 4.18e+03 | 3.36e+03 | 1.24 |
case6470_rte | 50482 | 75976 | 3.70e+03 | 3.24e+03 | 1.14 |
case6495_rte | 50426 | 76124 | 4.59e+03 | 3.70e+03 | 1.24 |
case6515_rte | 50546 | 76290 | 4.20e+03 | 3.40e+03 | 1.23 |
case7336_epigrids | 62116 | 95306 | 5.35e+03 | 5.04e+03 | 1.06 |
case8387_pegase | 78748 | 118702 | N.D. | N.D. | N.D. |
case9241_pegase | 85568 | 130826 | N.D. | N.D. | N.D. |
case9591_goc | 83572 | 130588 | N.D. | N.D. | N.D. |
case10000_goc | 76804 | 112352 | N.D. | N.D. | N.D. |
case10192_epigrids | 89850 | 139456 | N.D. | N.D. | N.D. |
case10480_goc | 96750 | 150874 | N.D. | N.D. | N.D. |
case13659_pegase | 117370 | 170588 | N.D. | N.D. | N.D. |
case19402_goc | 179562 | 281733 | N.D. | N.D. | N.D. |
case20758_epigrids | 179236 | 274918 | N.D. | N.D. | N.D. |
case24464_goc | 203374 | 313641 | N.D. | N.D. | N.D. |
case30000_goc | 208624 | 307752 | N.D. | N.D. | N.D. |
case78484_epigrids | 674562 | 1039062 | N.D. | N.D. | N.D. |
Thanks again @ccoffrin for the amount of work this represents.
This is very interesting. Do you have the logs of what failed in case1888_rte
? and in case8387_pegase
? I am suspecting you get an "out-of-memory" at some point !? So the conclusion seems to be that having the type-stable model as a default seems good.
Back to your primary question:
- What version of the NLPModels implementation would you like to be the default?
- Would you like any other version(s) to sick around as a non-default "variant" of NLPModels? The simple answer is: I don't really have any other suggestion :).
The reason being that this benchmark uses Ipopt so essentially it requires 3 derivatives from the model:
We are not (at the moment) investigating more ways to compute the sparse Jacobian/Hessian with automatic differentiation (even though one issue https://github.com/JuliaSmoothOptimizers/ADNLPModels.jl/issues/204 is directly linked to this topic), but more interested in ways to compute matrix-vector products directly with AD, i.e. without having to compute/evaluate the whole matrix. We do however give the possibility for the user to manually provide the sparsity pattern, which can save a ton of time for specific applications (like discretized PDE-constrained problems, where the sparsity pattern is essentially known).
Copied from Optimization example. I'll make a PR once CI is merged