search

jump-dev / ParametricOptInterface.jl

Extension for dealing with parameters
MIT License
36 stars 7 forks source link

Comparison of performances between POI and ParameterJuMP #41

Closed caioluke closed 2 years ago

caioluke commented 3 years ago

DISCUSSION

I have run an adapted version of this benchmark to compare the performances between POI and ParameterJuMP, a package that serves the same purpose as POI.

The benchmark in question is a Benders Decomposition applied to a Norm-1 regression.

The code used can be found here.

The tables below illustrate how long each approach (POI, ParameterJuMP, neither of them - which means only JuMP) was stuck doing which part of the BD algorithm.

While ParameterJuMP still performs better in the loop part, the gap between it and POI is not that large and it can (and hopefully will) be closed by POI.

On the other hand, POI allocates less memory than ParameterJuMP and has a faster initialization.

All in all, I think these results are nice for POI!

ParameterJuMP 
─────────────────────────────────────────────────────────────────────────
                                  Time                   Allocations
                          ──────────────────────   ───────────────────────
     Tot / % measured:          283s / 100%            27.8GiB / 100%
 Section          ncalls     time   %tot     avg     alloc   %tot      avg
 ─────────────────────────────────────────────────────────────────────────
 Loop                  1     239s  84.4%    239s   4.82GiB  17.3%  4.82GiB
   solve master       10     216s  76.3%   21.6s    106MiB  0.37%  10.6MiB
   solve nodes        10    19.9s  7.03%   1.99s   3.01GiB  10.8%   308MiB
     opt           10.0k    19.0s  6.72%  1.90ms   2.83GiB  10.2%   297KiB
     dual          10.0k    143ms  0.05%  14.3μs   77.5MiB  0.27%  7.94KiB
     fix           10.0k    135ms  0.05%  13.5μs   77.5MiB  0.27%  7.94KiB
   add cuts           10    3.26s  1.15%   326ms   1.70GiB  6.12%   174MiB
 Init                  1    44.1s  15.6%   44.1s   23.0GiB  82.7%  23.0GiB
   Slaves              1    42.3s  14.9%   42.3s   22.7GiB  81.7%  22.7GiB
   Cuts                1    1.83s  0.65%   1.83s    268MiB  0.94%   268MiB
     opt           1.00k    1.70s  0.60%  1.70ms    245MiB  0.86%   251KiB
     fix           1.00k   14.3ms  0.01%  14.3μs   7.75MiB  0.03%  7.94KiB
     dual          1.00k   14.2ms  0.01%  14.2μs   7.75MiB  0.03%  7.94KiB
   Master              1   4.30ms  0.00%  4.30ms    939KiB  0.00%   939KiB
   Sol                 1   1.70μs  0.00%  1.70μs   7.97KiB  0.00%  7.97KiB
 ─────────────────────────────────────────────────────────────────────────
POI
 ─────────────────────────────────────────────────────────────────────────
                                  Time                   Allocations
                          ──────────────────────   ───────────────────────
     Tot / % measured:          280s / 100%            22.6GiB / 100%
 Section          ncalls     time   %tot     avg     alloc   %tot      avg
 ─────────────────────────────────────────────────────────────────────────
 Loop                  1     258s  92.3%    258s   4.60GiB  20.4%  4.60GiB
   solve master       10     226s  80.8%   22.6s    106MiB  0.46%  10.6MiB
   solve nodes        10    28.3s  10.1%   2.83s   2.79GiB  12.4%   286MiB
     opt           10.0k    25.7s  9.19%  2.57ms   1.86GiB  8.25%   195KiB
     fix           10.0k    1.40s  0.50%   140μs    462MiB  2.00%  47.3KiB
     dual          10.0k    935ms  0.33%  93.5μs    460MiB  1.99%  47.1KiB
   add cuts           10    3.90s  1.39%   390ms   1.70GiB  7.53%   174MiB
 Init                  1    21.5s  7.68%   21.5s   18.0GiB  79.6%  18.0GiB
   Slaves              1    18.7s  6.67%   18.7s   17.6GiB  78.2%  17.6GiB
   Cuts                1    2.82s  1.01%   2.82s    342MiB  1.48%   342MiB
     opt           1.00k    2.52s  0.90%  2.52ms    149MiB  0.64%   152KiB
     fix           1.00k    148ms  0.05%   148μs    136MiB  0.59%   139KiB
     dual          1.00k   57.5ms  0.02%  57.5μs   46.0MiB  0.20%  47.1KiB
   Master              1   3.03ms  0.00%  3.03ms    939KiB  0.00%   939KiB
   Sol                 1   2.80μs  0.00%  2.80μs   7.97KiB  0.00%  7.97KiB
 ─────────────────────────────────────────────────────────────────────────
JuMP only
 ─────────────────────────────────────────────────────────────────────────
                                  Time                   Allocations
                          ──────────────────────   ───────────────────────
     Tot / % measured:          379s / 100%            67.3GiB / 100%
 Section          ncalls     time   %tot     avg     alloc   %tot      avg
 ─────────────────────────────────────────────────────────────────────────
 Loop                  1     336s  88.8%    336s   37.6GiB  55.9%  37.6GiB
   solve master       10     220s  58.1%   22.0s    106MiB  0.15%  10.6MiB
   solve nodes        10     113s  29.9%   11.3s   35.8GiB  53.3%  3.58GiB
     dual          10.0k    48.4s  12.8%  4.84ms   22.9GiB  34.0%  2.34MiB
     opt           10.0k    45.5s  12.0%  4.55ms    122MiB  0.18%  12.5KiB
     fix           10.0k    18.8s  4.97%  1.88ms   12.8GiB  19.0%  1.31MiB
   add cuts           10    2.94s  0.78%   294ms   1.70GiB  2.53%   174MiB
 Init                  1    42.5s  11.2%   42.5s   29.7GiB  44.1%  29.7GiB
   Slaves              1    30.4s  8.01%   30.4s   26.0GiB  38.7%  26.0GiB
   Cuts                1    12.2s  3.21%   12.2s   3.64GiB  5.40%  3.64GiB
     opt           1.00k    5.35s  1.41%  5.35ms   61.6MiB  0.09%  63.1KiB
     dual          1.00k    5.08s  1.34%  5.08ms   2.29GiB  3.40%  2.34MiB
     fix           1.00k    1.63s  0.43%  1.63ms   1.28GiB  1.90%  1.31MiB
   Master              1   3.14ms  0.00%  3.14ms    939KiB  0.00%   939KiB
   Sol                 1   2.70μs  0.00%  2.70μs   7.97KiB  0.00%  7.97KiB
jd-lara commented 3 years ago

Probably for larger systems POI will be even better due to the memory allocations.

guilhermebodin commented 2 years ago

replaced by #68