search

ami-iit / bipedal-locomotion-framework

Suite of libraries for achieving bipedal locomotion on humanoid robots
https://ami-iit.github.io/bipedal-locomotion-framework/
BSD 3-Clause "New" or "Revised" License
150 stars 38 forks source link

Investigate differences on TimeVaryingDCMPlannerUnitTests execution time depending on dependency installation method #241

Open traversaro opened 3 years ago

traversaro commented 3 years ago

As a follow up for https://github.com/dic-iit/bipedal-locomotion-framework/issues/219, it turns out that installing ipopt+mumps via coinbrew is now sensibly faster (up to 10x) over other alternatives. In this issue, we will investigate why this is the case, as the coinbrew installation is a non-standard step that complexifies deployment.

To have a reproducible environment, I run the tests in a dedicated GitHub Action workflow. Even if GitHub Action machines can change overtime and will not reflect exactly the hardware in which we run the walking controller on the real robot, from preliminary investigation they were able to reproduce the macroscopic performance differences in https://github.com/dic-iit/bipedal-locomotion-framework/issues/219, so it is worth to use it for the time being.

traversaro commented 3 years ago

In https://github.com/traversaro/ipopt-walking-benchmarks/runs/2133731892 I collected the first results, with five different jobs. The tests are run 10 times to ensure that the variance across runs is not impacting the analysis.

Name Time spent executing TimeVaryingDCMPlannerUnitTests (s) Description
conda-manual 2.1 All dependencies installed via conda (conda-forge, robotology-staging), bfl compiled manually
conda-superbuild 2.3 mumps, ipopt installed via conda (conda-forge), casadi and bfl installed via superbuild
conda-coinbrew-superbuild 0.25 mumps, ipopt installed via coinbrew (conda-forge), casadi and bfl installed via superbuild with conda-forge deps
apt-superbuild 0.8 mumps, ipopt installed via apt, casadi and blf sia superbuild
apt-coinbrew-superbuild 0.21 mumps, ipopt installed via coinbrew, casadi and blf sia superbuild with apt deps
traversaro commented 3 years ago

The nice news is that the major speedup obtained by using coinbrew is there for both apt and conda, so it could be related to the use of mumps 4 (apt and conda all use mumps 5.2.1). Interestingly, the coinbrew builds are associated with a low number of iterations of the NLP problem (see the following outputs), so the issue it not simply that mumps 4 is faster, it is really being called a different number of times.

conda-manual

2021-03-17T19:02:16.3325145Z     Start 17: TimeVaryingDCMPlannerUnitTests
2021-03-17T19:02:16.3325933Z 
2021-03-17T19:02:16.3328335Z 17: Test command: /home/runner/work/ipopt-walking-benchmarks/ipopt-walking-benchmarks/bipedal-locomotion-framework/build/bin/TimeVaryingDCMPlannerUnitTests
2021-03-17T19:02:16.3330091Z 17: Test timeout computed to be: 1500
2021-03-17T19:02:16.3392688Z 17: [2021-03-17 19:02:16.338] [blf] [error] [StdImplementation::getParameterPrivate] Parameter named 'linear_solver' not found.
2021-03-17T19:02:16.3395063Z 17: [2021-03-17 19:02:16.338] [blf] [info] [TimeVaryingDCMPlanner::initialize] linear_solver not found. The following parameter will be used mumps.
2021-03-17T19:02:16.3398647Z 17: [2021-03-17 19:02:16.338] [blf] [error] [StdImplementation::getParameterPrivate] Parameter named 'use_external_dcm_reference' not found.
2021-03-17T19:02:16.3404653Z 17: [2021-03-17 19:02:16.338] [blf] [info] [TimeVaryingDCMPlanner::initialize] use_external_dcm_reference not found. The following parameter will be used false.
2021-03-17T19:02:16.3407116Z 17: [2021-03-17 19:02:16.338] [blf] [error] [StdImplementation::getParameterPrivate] Parameter named 'gravity' not found.
2021-03-17T19:02:16.3408952Z 17: [2021-03-17 19:02:16.338] [blf] [info] [TimeVaryingDCMPlanner::initialize] gravity not found. The following parameter will be used 9.80665.
2021-03-17T19:02:16.4831375Z 17: 
2021-03-17T19:02:16.4832258Z 17: ******************************************************************************
2021-03-17T19:02:16.4835252Z 17: This program contains Ipopt, a library for large-scale nonlinear optimization.
2021-03-17T19:02:16.4836618Z 17:  Ipopt is released as open source code under the Eclipse Public License (EPL).
2021-03-17T19:02:16.4837975Z 17:          For more information visit https://github.com/coin-or/Ipopt
2021-03-17T19:02:16.4838863Z 17: ******************************************************************************
2021-03-17T19:02:16.4839414Z 17: 
2021-03-17T19:02:18.4199102Z 17:       solver  :   t_proc      (avg)   t_wall      (avg)    n_eval
2021-03-17T19:02:18.4200425Z 17:        nlp_f  |   1.24ms ( 21.02us)   1.25ms ( 21.11us)        59
2021-03-17T19:02:18.4201323Z 17:        nlp_g  |   2.07ms ( 35.05us)   2.00ms ( 33.87us)        59
2021-03-17T19:02:18.4202014Z 17:     nlp_grad  | 109.00us (109.00us) 109.10us (109.10us)         1
2021-03-17T19:02:18.4202751Z 17:   nlp_grad_f  |   1.12ms ( 46.71us)   1.05ms ( 43.72us)        24
2021-03-17T19:02:18.4203454Z 17:   nlp_hess_l  |   1.71ms ( 85.60us)   1.65ms ( 82.31us)        20
2021-03-17T19:02:18.4204149Z 17:    nlp_jac_g  |   2.10ms ( 80.77us)   1.99ms ( 76.42us)        26
2021-03-17T19:02:18.4204808Z 17:        total  |   2.06 s (  2.06 s)   1.97 s (  1.97 s)         1
2021-03-17T19:02:18.4282892Z 17: ===============================================================================
2021-03-17T19:02:18.4283799Z 17: All tests passed (157 assertions in 1 test case)
2021-03-17T19:02:18.4284412Z 17: 
2021-03-17T19:02:18.4301747Z     Test #17: TimeVaryingDCMPlannerUnitTests ...   Passed    2.10 sec

conda-superbuild

2021-03-17T19:28:54.5940652Z     Start 17: TimeVaryingDCMPlannerUnitTests
2021-03-17T19:28:54.5942230Z 
2021-03-17T19:28:54.5945279Z 17: Test command: /home/runner/work/ipopt-walking-benchmarks/ipopt-walking-benchmarks/robotology-superbuild/build/src/bipedal-locomotion-framework/bin/TimeVaryingDCMPlannerUnitTests
2021-03-17T19:28:54.5947548Z 17: Test timeout computed to be: 1500
2021-03-17T19:28:54.5998799Z 17: [2021-03-17 19:28:54.599] [blf] [error] [StdImplementation::getParameterPrivate] Parameter named 'linear_solver' not found.
2021-03-17T19:28:54.6001279Z 17: [2021-03-17 19:28:54.599] [blf] [info] [TimeVaryingDCMPlanner::initialize] linear_solver not found. The following parameter will be used mumps.
2021-03-17T19:28:54.6003041Z 17: [2021-03-17 19:28:54.599] [blf] [error] [StdImplementation::getParameterPrivate] Parameter named 'use_external_dcm_reference' not found.
2021-03-17T19:28:54.6004839Z 17: [2021-03-17 19:28:54.599] [blf] [info] [TimeVaryingDCMPlanner::initialize] use_external_dcm_reference not found. The following parameter will be used false.
2021-03-17T19:28:54.6007519Z 17: [2021-03-17 19:28:54.599] [blf] [error] [StdImplementation::getParameterPrivate] Parameter named 'gravity' not found.
2021-03-17T19:28:54.6009758Z 17: [2021-03-17 19:28:54.599] [blf] [info] [TimeVaryingDCMPlanner::initialize] gravity not found. The following parameter will be used 9.80665.
2021-03-17T19:28:54.7339348Z 17: 
2021-03-17T19:28:54.7340344Z 17: ******************************************************************************
2021-03-17T19:28:54.7341909Z 17: This program contains Ipopt, a library for large-scale nonlinear optimization.
2021-03-17T19:28:54.7342950Z 17:  Ipopt is released as open source code under the Eclipse Public License (EPL).
2021-03-17T19:28:54.7344317Z 17:          For more information visit https://github.com/coin-or/Ipopt
2021-03-17T19:28:54.7345148Z 17: ******************************************************************************
2021-03-17T19:28:54.7345833Z 17: 
2021-03-17T19:28:56.9383163Z 17:       solver  :   t_proc      (avg)   t_wall      (avg)    n_eval
2021-03-17T19:28:56.9384591Z 17:        nlp_f  |   1.62ms ( 21.34us)   1.59ms ( 20.90us)        76
2021-03-17T19:28:56.9385432Z 17:        nlp_g  |   2.41ms ( 31.75us)   2.32ms ( 30.50us)        76
2021-03-17T19:28:56.9386129Z 17:     nlp_grad  | 113.00us (113.00us) 113.40us (113.40us)         1
2021-03-17T19:28:56.9387033Z 17:   nlp_grad_f  |   1.05ms ( 42.04us) 988.70us ( 39.55us)        25
2021-03-17T19:28:56.9388473Z 17:   nlp_hess_l  |   1.90ms ( 90.33us)   1.78ms ( 84.99us)        21
2021-03-17T19:28:56.9389145Z 17:    nlp_jac_g  |   2.09ms ( 77.33us)   2.02ms ( 74.90us)        27
2021-03-17T19:28:56.9389810Z 17:        total  |   2.32 s (  2.32 s)   2.24 s (  2.24 s)         1
2021-03-17T19:28:56.9462397Z 17: ===============================================================================
2021-03-17T19:28:56.9463337Z 17: All tests passed (157 assertions in 1 test case)
2021-03-17T19:28:56.9464054Z 17: 
2021-03-17T19:28:56.9477507Z 1/1 Test #17: TimeVaryingDCMPlannerUnitTests ...   Passed    2.35 sec

conda-coinbrew-superbuild

2021-03-17T19:31:48.3176155Z     Start 17: TimeVaryingDCMPlannerUnitTests
2021-03-17T19:31:48.3176699Z 
2021-03-17T19:31:48.3179032Z 17: Test command: /home/runner/work/ipopt-walking-benchmarks/ipopt-walking-benchmarks/robotology-superbuild/build/src/bipedal-locomotion-framework/bin/TimeVaryingDCMPlannerUnitTests
2021-03-17T19:31:48.3180543Z 17: Test timeout computed to be: 1500
2021-03-17T19:31:48.3243458Z 17: [2021-03-17 19:31:48.324] [blf] [error] [StdImplementation::getParameterPrivate] Parameter named 'linear_solver' not found.
2021-03-17T19:31:48.3245358Z 17: [2021-03-17 19:31:48.324] [blf] [info] [TimeVaryingDCMPlanner::initialize] linear_solver not found. The following parameter will be used mumps.
2021-03-17T19:31:48.3247503Z 17: [2021-03-17 19:31:48.324] [blf] [error] [StdImplementation::getParameterPrivate] Parameter named 'use_external_dcm_reference' not found.
2021-03-17T19:31:48.3249699Z 17: [2021-03-17 19:31:48.324] [blf] [info] [TimeVaryingDCMPlanner::initialize] use_external_dcm_reference not found. The following parameter will be used false.
2021-03-17T19:31:48.3251848Z 17: [2021-03-17 19:31:48.324] [blf] [error] [StdImplementation::getParameterPrivate] Parameter named 'gravity' not found.
2021-03-17T19:31:48.3253419Z 17: [2021-03-17 19:31:48.324] [blf] [info] [TimeVaryingDCMPlanner::initialize] gravity not found. The following parameter will be used 9.80665.
2021-03-17T19:31:48.4245437Z 17: 
2021-03-17T19:31:48.4246241Z 17: ******************************************************************************
2021-03-17T19:31:48.4247551Z 17: This program contains Ipopt, a library for large-scale nonlinear optimization.
2021-03-17T19:31:48.4248524Z 17:  Ipopt is released as open source code under the Eclipse Public License (EPL).
2021-03-17T19:31:48.4249807Z 17:          For more information visit https://github.com/coin-or/Ipopt
2021-03-17T19:31:48.4250714Z 17: ******************************************************************************
2021-03-17T19:31:48.4251258Z 17: 
2021-03-17T19:31:48.5524545Z 17:       solver  :   t_proc      (avg)   t_wall      (avg)    n_eval
2021-03-17T19:31:48.5525348Z 17:        nlp_f  | 349.00us ( 19.39us) 295.01us ( 16.39us)        18
2021-03-17T19:31:48.5525904Z 17:        nlp_g  | 587.00us ( 32.61us) 487.21us ( 27.07us)        18
2021-03-17T19:31:48.5526969Z 17:     nlp_grad  | 158.00us (158.00us) 158.30us (158.30us)         1
2021-03-17T19:31:48.5527963Z 17:   nlp_grad_f  | 644.00us ( 33.89us) 531.41us ( 27.97us)        19
2021-03-17T19:31:48.5528599Z 17:   nlp_hess_l  |   1.26ms ( 73.82us)   1.05ms ( 61.99us)        17
2021-03-17T19:31:48.5529198Z 17:    nlp_jac_g  |   1.17ms ( 61.53us) 990.01us ( 52.11us)        19
2021-03-17T19:31:48.5529957Z 17:        total  | 188.32ms (188.32ms) 141.30ms (141.30ms)         1
2021-03-17T19:31:48.5591185Z 17: ===============================================================================
2021-03-17T19:31:48.5592397Z 17: All tests passed (157 assertions in 1 test case)
2021-03-17T19:31:48.5592958Z 17: 
2021-03-17T19:31:48.5608202Z 1/1 Test #17: TimeVaryingDCMPlannerUnitTests ...   Passed    0.24 sec

apt-superbuild

2021-03-17T19:27:06.7945406Z     Test #17: TimeVaryingDCMPlannerUnitTests ...   Passed    0.74 sec
2021-03-17T19:27:06.7948484Z     Start 17: TimeVaryingDCMPlannerUnitTests
2021-03-17T19:27:06.7949311Z 
2021-03-17T19:27:06.7951456Z 17: Test command: /home/runner/work/ipopt-walking-benchmarks/ipopt-walking-benchmarks/robotology-superbuild/build/src/bipedal-locomotion-framework/bin/TimeVaryingDCMPlannerUnitTests
2021-03-17T19:27:06.7953067Z 17: Test timeout computed to be: 1500
2021-03-17T19:27:06.8005215Z 17: [2021-03-17 19:27:06.800] [blf] [error] [StdImplementation::getParameterPrivate] Parameter named 'linear_solver' not found.
2021-03-17T19:27:06.8007125Z 17: [2021-03-17 19:27:06.800] [blf] [info] [TimeVaryingDCMPlanner::initialize] linear_solver not found. The following parameter will be used mumps.
2021-03-17T19:27:06.8009891Z 17: [2021-03-17 19:27:06.800] [blf] [error] [StdImplementation::getParameterPrivate] Parameter named 'use_external_dcm_reference' not found.
2021-03-17T19:27:06.8012403Z 17: [2021-03-17 19:27:06.800] [blf] [info] [TimeVaryingDCMPlanner::initialize] use_external_dcm_reference not found. The following parameter will be used false.
2021-03-17T19:27:06.8014302Z 17: [2021-03-17 19:27:06.800] [blf] [error] [StdImplementation::getParameterPrivate] Parameter named 'gravity' not found.
2021-03-17T19:27:06.8016230Z 17: [2021-03-17 19:27:06.800] [blf] [info] [TimeVaryingDCMPlanner::initialize] gravity not found. The following parameter will be used 9.80665.
2021-03-17T19:27:06.9355595Z 17: 
2021-03-17T19:27:06.9356387Z 17: ******************************************************************************
2021-03-17T19:27:06.9359390Z 17: This program contains Ipopt, a library for large-scale nonlinear optimization.
2021-03-17T19:27:06.9360328Z 17:  Ipopt is released as open source code under the Eclipse Public License (EPL).
2021-03-17T19:27:06.9361535Z 17:          For more information visit http://projects.coin-or.org/Ipopt
2021-03-17T19:27:06.9362416Z 17: ******************************************************************************
2021-03-17T19:27:06.9363435Z 17: 
2021-03-17T19:27:07.5875457Z 17:       solver  :   t_proc      (avg)   t_wall      (avg)    n_eval
2021-03-17T19:27:07.5876017Z 17:        nlp_f  |   1.09ms ( 18.46us)   1.07ms ( 18.22us)        59
2021-03-17T19:27:07.5876486Z 17:        nlp_g  |   1.63ms ( 27.56us)   1.63ms ( 27.55us)        59
2021-03-17T19:27:07.5876942Z 17:     nlp_grad  |  93.00us ( 93.00us)  92.70us ( 92.70us)         1
2021-03-17T19:27:07.5877946Z 17:   nlp_grad_f  |   1.00ms ( 32.42us)   1.01ms ( 32.52us)        31
2021-03-17T19:27:07.5878440Z 17:   nlp_hess_l  |   2.08ms ( 74.14us)   2.08ms ( 74.46us)        28
2021-03-17T19:27:07.5878924Z 17:    nlp_jac_g  |   2.18ms ( 65.94us)   2.19ms ( 66.22us)        33
2021-03-17T19:27:07.5879383Z 17:        total  | 670.77ms (670.77ms) 671.02ms (671.02ms)         1
2021-03-17T19:27:07.5953730Z 17: ===============================================================================
2021-03-17T19:27:07.5954485Z 17: All tests passed (157 assertions in 1 test case)
2021-03-17T19:27:07.5954930Z 17: 
2021-03-17T19:27:07.5970145Z     Test #17: TimeVaryingDCMPlannerUnitTests ...   Passed    0.80 sec

apt-coinbrew-superbuild

2021-03-17T19:26:49.9798344Z     Start 17: TimeVaryingDCMPlannerUnitTests
2021-03-17T19:26:49.9798878Z 
2021-03-17T19:26:49.9801082Z 17: Test command: /home/runner/work/ipopt-walking-benchmarks/ipopt-walking-benchmarks/robotology-superbuild/build/src/bipedal-locomotion-framework/bin/TimeVaryingDCMPlannerUnitTests
2021-03-17T19:26:49.9802853Z 17: Test timeout computed to be: 1500
2021-03-17T19:26:49.9855886Z 17: [2021-03-17 19:26:49.985] [blf] [error] [StdImplementation::getParameterPrivate] Parameter named 'linear_solver' not found.
2021-03-17T19:26:49.9857611Z 17: [2021-03-17 19:26:49.985] [blf] [info] [TimeVaryingDCMPlanner::initialize] linear_solver not found. The following parameter will be used mumps.
2021-03-17T19:26:49.9859384Z 17: [2021-03-17 19:26:49.985] [blf] [error] [StdImplementation::getParameterPrivate] Parameter named 'use_external_dcm_reference' not found.
2021-03-17T19:26:49.9861198Z 17: [2021-03-17 19:26:49.985] [blf] [info] [TimeVaryingDCMPlanner::initialize] use_external_dcm_reference not found. The following parameter will be used false.
2021-03-17T19:26:49.9862914Z 17: [2021-03-17 19:26:49.985] [blf] [error] [StdImplementation::getParameterPrivate] Parameter named 'gravity' not found.
2021-03-17T19:26:49.9864536Z 17: [2021-03-17 19:26:49.985] [blf] [info] [TimeVaryingDCMPlanner::initialize] gravity not found. The following parameter will be used 9.80665.
2021-03-17T19:26:50.0902651Z 17: 
2021-03-17T19:26:50.0903704Z 17: ******************************************************************************
2021-03-17T19:26:50.0905936Z 17: This program contains Ipopt, a library for large-scale nonlinear optimization.
2021-03-17T19:26:50.0907644Z 17:  Ipopt is released as open source code under the Eclipse Public License (EPL).
2021-03-17T19:26:50.0909637Z 17:          For more information visit https://github.com/coin-or/Ipopt
2021-03-17T19:26:50.0910807Z 17: ******************************************************************************
2021-03-17T19:26:50.0911444Z 17: 
2021-03-17T19:26:50.1774741Z 17:       solver  :   t_proc      (avg)   t_wall      (avg)    n_eval
2021-03-17T19:26:50.1775749Z 17:        nlp_f  | 407.00us ( 22.61us) 408.52us ( 22.70us)        18
2021-03-17T19:26:50.1776566Z 17:        nlp_g  | 474.00us ( 26.33us) 470.43us ( 26.13us)        18
2021-03-17T19:26:50.1777394Z 17:     nlp_grad  | 109.00us (109.00us) 108.51us (108.51us)         1
2021-03-17T19:26:50.1778264Z 17:   nlp_grad_f  | 684.00us ( 36.00us) 684.94us ( 36.05us)        19
2021-03-17T19:26:50.1779118Z 17:   nlp_hess_l  | 968.00us ( 56.94us) 966.06us ( 56.83us)        17
2021-03-17T19:26:50.1779977Z 17:    nlp_jac_g  |   1.01ms ( 52.89us)   1.01ms ( 53.05us)        19
2021-03-17T19:26:50.1780795Z 17:        total  |  99.68ms ( 99.68ms)  99.70ms ( 99.70ms)         1
2021-03-17T19:26:50.1833321Z 17: ===============================================================================
2021-03-17T19:26:50.1834346Z 17: All tests passed (157 assertions in 1 test case)
2021-03-17T19:26:50.1835194Z 17: 
2021-03-17T19:26:50.1842930Z     Test #17: TimeVaryingDCMPlannerUnitTests ...   Passed    0.20 sec
GiulioRomualdi commented 3 years ago

It would be interesting also to run the test with pardiso. Accordingly to the documentation ipopt can be used with pardiso https://coin-or.github.io/Ipopt/INSTALL.html#DOWNLOAD_PARDISO

However I was not able to use it since pardiso seems to be incompatible with the lapack/blas installation provided in ubuntu. Indeed when I compile coinbrew with

./coinbrew build Ipopt --prefix=install --with-pardiso="$HOME/robot-install/lib/libpardiso600-GNU800-X86-64.so -fopenmp -lgfortran" --test --no-prompt --verbosity=3

I got the following errors

configure:17101: checking for function pardiso_ipopt_newinterface_ in -llapack -lblas /home/gromualdi/robot-install/lib/libpardiso600-GNU800-X86-64.so -fopenmp -lgfortran  
configure:17129: gcc -o conftest -O2 -DNDEBUG    conftest.c -llapack -lblas /home/gromualdi/robot-install/lib/libpardiso600-GNU800-X86-64.so -fopenmp -lgfortran   >&5
/usr/bin/ld: /home/gromualdi/robot-install/lib/libpardiso600-GNU800-X86-64.so: undefined reference to `dtrmm_'
/usr/bin/ld: /home/gromualdi/robot-install/lib/libpardiso600-GNU800-X86-64.so: undefined reference to `lsame_'
/usr/bin/ld: /home/gromualdi/robot-install/lib/libpardiso600-GNU800-X86-64.so: undefined reference to `dlartg_'
/usr/bin/ld: /home/gromualdi/robot-install/lib/libpardiso600-GNU800-X86-64.so: undefined reference to `dswap_'

A possible solution is to switch to MKL intel library. I tried but the migration is not straightforward

We may use this: https://github.com/eddelbuettel/mkl4deb

traversaro commented 3 years ago

Just FYI, as conda + coinbrew is approximately as fast as apt + coinbrew, a possible option is also to install mkl via conda-forge, see https://conda-forge.org/docs/maintainer/knowledge_base.html#switching-blas-implementation .

GiulioRomualdi commented 3 years ago

I tried to run the same test with snopt. The time performances follow.

TLDR. IPOPT is faster than SNOPT

 ==============================
    SNOPT  C interface  2.2.0   
 ==============================
 S N O P T  7.7.7    (Feb 2021)
 ==============================

 SNMEMB EXIT 100 -- finished successfully
 SNMEMB INFO 104 -- memory requirements estimated

 Trial version of SNOPT -- for evaluation or academic purposes only

 Nonlinear constraints    1697     Linear constraints       0
 Nonlinear variables      1124     Linear variables         0
 Jacobian  variables      1124     Objective variables   1124
 Total constraints        1697     Total variables       1124

 The user has defined    4695   out of    4695   constraint gradients.
 The user has defined    1124   out of    1124   objective  gradients.

        Minor NonOpt  QP mult  QP step   rgNorm Elastic QP obj     nS
          100     68  2.0E-01  1.0E+00  2.9E-11  5.3731455E+03     92

 Major Minors     Step   nCon Feasible  Optimal  MeritFunction     nS Penalty
     0    107               1  1.7E-01  7.5E-01  1.3202648E-01     99           r iT

        Minor NonOpt  QP mult  QP step   rgNorm Elastic QP obj     nS
          100     26 -2.1E-01  1.0E+00  4.0E-13 -2.6961898E+02    180

 Major Minors     Step   nCon Feasible  Optimal  MeritFunction     nS Penalty
     1    118  1.0E+00      2  8.2E-03  3.5E-02  4.5688665E+02    198 3.0E+03   r iT
     2     35  4.1E-01      4  4.9E-03  2.1E-02  2.4680235E+02    226 1.5E+02 s   i
     3     16  1.0E+00      5  8.9E-06  5.0E-04  1.6299024E+02    238 6.9E+01     i
     4     12  1.0E+00      6  3.1E-06  3.3E-04  1.2835157E+02    245 6.9E+01     i
     5      5  1.0E+00      7  5.0E-06  2.6E-04  1.0482914E+02    249 6.9E+01     i
     6      8  1.0E+00      8  2.3E-06  1.7E-04  9.2158457E+01    252 6.9E+01     i
     7      4  1.0E+00      9  3.7E-06  1.3E-04  8.2040395E+01    253 6.9E+01     i
     8      6  1.0E+00     10  3.5E-06  1.3E-04  7.5189058E+01    256 6.9E+01     i
     9      9  1.0E+00     11  2.7E-06  1.1E-04  6.9609409E+01    256 6.9E+01     i

 Major Minors     Step   nCon Feasible  Optimal  MeritFunction     nS Penalty
    10      6  1.0E+00     12  1.9E-06  1.1E-04  6.4988621E+01    259 6.9E+01     i
    11      9  1.0E+00     13  1.8E-06  1.0E-04  6.0798673E+01    259 6.9E+01     i
    12      4  1.0E+00     14  1.6E-06  9.3E-05  5.7302221E+01    262 6.9E+01   R i
    13      5  1.0E+00     15 (6.9E-08) 6.2E-05  5.5947757E+01    264 6.9E+01 s   i
    14      3  1.0E+00     16  1.0E-06  1.5E-04  5.5268399E+01    266 6.9E+01     i
    15      4  1.0E+00     17 (3.6E-07) 7.2E-05  5.3862150E+01    269 6.9E+01     i
    16      2  1.0E+00     18 (2.1E-07) 6.2E-05  5.3474768E+01    270 6.9E+01     i
    17      3  1.0E+00     19 (3.2E-07) 2.9E-05  5.2699865E+01    272 6.9E+01     i
    18      4  1.0E+00     20  2.0E-06  8.5E-05  5.2179207E+01    275 6.9E+01     i
    19      5  1.0E+00     21  2.1E-06  4.4E-05  5.1431516E+01    279 6.9E+01     i

 Major Minors     Step   nCon Feasible  Optimal  MeritFunction     nS Penalty
    20      2  1.0E+00     22  2.1E-06  6.6E-05  5.1108981E+01    280 6.9E+01     i
    21      4  1.0E+00     23 (6.1E-07) 3.3E-05  5.0595279E+01    281 6.9E+01     i
    22      4  1.0E+00     24  2.0E-06  8.1E-05  5.0411760E+01    284 6.9E+01     i
    23      3  1.0E+00     25 (8.8E-07) 2.7E-05  4.9891055E+01    286 6.9E+01   R i
    24      2  4.9E-01     27 (4.6E-07) 1.7E-05  4.9767471E+01    287 6.9E+01 s   i
    25      3  1.0E+00     28 (5.0E-08) 1.2E-05  4.9695105E+01    289 6.9E+01     i
    26      2  1.0E+00     29 (2.5E-07) 1.8E-05  4.9538009E+01    290 6.9E+01     i
    27      3  1.0E+00     30 (9.1E-07) 1.7E-05  4.9354306E+01    292 6.9E+01     i
    28      2  1.0E+00     31 (5.8E-07) 1.6E-05  4.9244687E+01    293 6.9E+01     i
    29      3  1.0E+00     32 (7.5E-07) 1.3E-05  4.9131183E+01    295 6.9E+01     i

 Major Minors     Step   nCon Feasible  Optimal  MeritFunction     nS Penalty
    30      3  1.0E+00     33  1.2E-06  1.8E-05  4.8987378E+01    297 6.9E+01     i
    31      3  1.0E+00     34  2.2E-06  1.9E-05  4.8821287E+01    299 6.9E+01     i
    32      4  1.0E+00     35  1.7E-06  2.2E-05  4.8690443E+01    302 6.9E+01     i
    33      4  1.0E+00     36  1.4E-06  1.9E-05  4.8571840E+01    305 6.9E+01     i
    34      1  1.0E+00     37  2.5E-06  2.3E-05  4.8409817E+01    305 6.9E+01   R i
    35      2  1.0E+00     38 (5.0E-08) 1.5E-05  4.8310697E+01    306 6.9E+01 s   i
    36      3  1.0E+00     39 (1.1E-07) 1.7E-05  4.8175335E+01    308 6.9E+01     i
    37      3  1.0E+00     40 (1.2E-07) 1.1E-05  4.8076262E+01    310 6.9E+01     i
    38      3  1.0E+00     41 (9.5E-08) 1.3E-05  4.8021922E+01    312 6.9E+01     i
    39      3  1.0E+00     42 (2.1E-07) 1.2E-05  4.7952149E+01    314 6.9E+01     i

 Major Minors     Step   nCon Feasible  Optimal  MeritFunction     nS Penalty
    40      3  1.0E+00     43 (5.0E-07) 1.4E-05  4.7860916E+01    316 6.9E+01     i
    41      3  1.0E+00     44  1.5E-06  9.8E-06  4.7787007E+01    318 6.9E+01     i
    42      3  1.0E+00     45 (6.3E-08) 9.8E-06  4.7739212E+01    320 6.9E+01     i
    43      4  1.0E+00     46 (7.0E-08) 1.2E-05  4.7695749E+01    323 6.9E+01     i
    44      4  1.0E+00     47  1.0E-06  1.7E-05  4.7589261E+01    326 6.9E+01     i
    45      3  1.0E+00     48  1.4E-05  6.1E-05  4.7359543E+01    328 6.9E+01   R i
    46      2  1.0E+00     49 (5.0E-08) 2.1E-05  4.7310566E+01    329 6.9E+01 s   i
    47      1  1.0E+00     50 (5.0E-08) 8.9E-06  4.7225612E+01    329 6.9E+01     i
    48      2  1.0E+00     51 (5.0E-08) 5.7E-06  4.7196546E+01    330 6.9E+01     i
    49      4  1.0E+00     52 (5.0E-08) 6.7E-06  4.7168462E+01    331 6.9E+01     i

 Major Minors     Step   nCon Feasible  Optimal  MeritFunction     nS Penalty
    50      3  1.0E+00     53 (5.0E-08) 6.4E-06  4.7148276E+01    333 6.9E+01     i
    51      2  1.0E+00     54 (8.2E-08) 6.1E-06  4.7124130E+01    334 6.9E+01     i
    52      3  1.0E+00     55 (1.6E-07) 7.4E-06  4.7094390E+01    336 6.9E+01     i
    53      2  1.0E+00     56 (3.8E-07) 9.6E-06  4.7054242E+01    337 6.9E+01     i
    54      4  1.0E+00     57 (6.8E-07) 1.2E-05  4.7005868E+01    340 6.9E+01     i
    55      2  1.0E+00     58 (9.3E-07) 1.3E-05  4.6953808E+01    341 6.9E+01     i
    56      2  1.0E+00     59 (9.5E-07) 1.2E-05  4.6902983E+01    342 6.9E+01   R i
    57      3  1.0E+00     60 (5.0E-08) 9.9E-06  4.6877156E+01    344 6.9E+01 s   i
    58      4  3.8E-01     62 (8.7E-08) 9.4E-06  4.6827941E+01    345 6.9E+01     i
    59      4  1.0E+00     63 (1.5E-07) 1.2E-05  4.6792525E+01    348 6.9E+01     i

 Major Minors     Step   nCon Feasible  Optimal  MeritFunction     nS Penalty
    60      2  4.8E-01     65 (9.8E-08) 9.9E-06  4.6772457E+01    349 6.9E+01     i
    61      4  1.0E+00     66 (1.4E-07) 1.5E-05  4.6758426E+01    352 6.9E+01     i
    62      5  1.0E+00     67 (8.3E-08) 1.0E-05  4.6714529E+01    354 6.9E+01     i
    63      3  3.8E-01     69 (1.2E-07) 8.3E-06  4.6688549E+01    355 6.9E+01     i
    64      4  1.0E+00     70 (5.6E-07) 1.1E-05  4.6657590E+01    356 6.9E+01     i
    65      2  1.0E+00     71  2.9E-06  1.9E-05  4.6626221E+01    357 6.9E+01     i
    66      4  1.0E+00     72  3.7E-06  1.9E-05  4.6579632E+01    358 6.9E+01     i
    67      7  1.0E+00     73 (1.5E-07) 2.0E-05  4.6542531E+01    356 6.9E+01   R i
    68      6  2.4E-01     75 (1.2E-07) 8.2E-06  4.6475539E+01    361 6.9E+01 s   i
    69      2  1.0E+00     76 (5.0E-08) 6.0E-06  4.6462063E+01    362 6.9E+01     i

 Major Minors     Step   nCon Feasible  Optimal  MeritFunction     nS Penalty
    70      2  1.0E+00     77 (5.0E-08) 6.0E-06  4.6440708E+01    361 6.9E+01     i
    71      2  1.0E+00     78 (5.0E-08) 1.1E-05  4.6418381E+01    360 6.9E+01     i
    72      1  1.0E+00     79 (7.8E-08) 1.4E-05  4.6388030E+01    360 6.9E+01     i
    73      1  1.0E+00     80 (8.8E-08) 1.0E-05  4.6362307E+01    360 6.9E+01     i
    74      2  1.0E+00     81 (5.8E-08) 4.9E-06  4.6344485E+01    361 6.9E+01     i
    75      1  1.0E+00     82 (5.0E-08) 4.1E-06  4.6332631E+01    361 6.9E+01     i
    76      1  1.0E+00     83 (5.0E-08) 7.1E-06  4.6321712E+01    361 6.9E+01     i
    77      1  1.0E+00     84 (5.7E-08) 8.4E-06  4.6309925E+01    361 6.9E+01     i
    78      2  1.0E+00     85 (1.0E-07) 8.0E-06  4.6297157E+01    362 6.9E+01   R i
    79      4  1.0E+00     86 (5.0E-08) 5.3E-06  4.6290620E+01    361 6.9E+01 s   i

 Major Minors     Step   nCon Feasible  Optimal  MeritFunction     nS Penalty
    80      5  1.0E+00     87 (6.9E-08) 7.4E-06  4.6278129E+01    361 6.9E+01     i
    81      1  5.1E-01     89 (5.8E-08) 6.7E-06  4.6266755E+01    361 6.9E+01     i
    82      2  1.0E+00     90 (5.7E-08) 1.2E-05  4.6260286E+01    362 6.9E+01     i
    83      2  1.0E+00     91 (8.1E-08) 7.1E-06  4.6239879E+01    361 6.9E+01     i
    84      2  5.0E-01     93 (2.2E-07) 5.5E-06  4.6220912E+01    362 6.9E+01     i
    85      4  1.0E+00     94  1.1E-06  1.1E-05  4.6204562E+01    363 6.9E+01     i
    86      4  1.0E+00     95  1.7E-06  8.3E-06  4.6173533E+01    366 6.9E+01     i
    87      3  4.9E-01     97  1.4E-06  7.5E-06  4.6148881E+01    366 6.9E+01     i
    88      4  1.0E+00     98 (9.3E-07) 1.0E-05  4.6117182E+01    365 6.9E+01     i
    89      5  1.0E+00     99  8.0E-06  3.4E-05  4.6018092E+01    367 6.9E+01   R i

 Major Minors     Step   nCon Feasible  Optimal  MeritFunction     nS Penalty
    90      3  2.9E-01    101  5.6E-06  2.4E-05  4.5977098E+01    369 6.9E+01 s   i
    91      2  1.0E+00    102 (5.0E-08) 3.2E-06  4.5968123E+01    370 6.9E+01     i
    92      2  1.0E+00    103 (5.0E-08) 2.6E-06  4.5960447E+01    371 6.9E+01     i
    93      1  1.0E+00    104 (5.0E-08) 3.4E-06  4.5953697E+01    371 6.9E+01     i
    94      1  1.0E+00    105 (5.0E-08) 4.9E-06  4.5945428E+01    371 6.9E+01     i
    95      1  1.0E+00    106 (5.0E-08) 5.4E-06  4.5937544E+01    371 6.9E+01     i
    96      1  1.0E+00    107 (5.0E-08) 4.7E-06  4.5930636E+01    371 6.9E+01     i
    97      1  1.0E+00    108 (5.0E-08) 2.8E-06  4.5925072E+01    371 6.9E+01     i
    98      1  1.0E+00    109 (5.0E-08) 2.3E-06  4.5920385E+01    371 6.9E+01     i
    99      2  1.0E+00    110 (5.0E-08) 2.8E-06  4.5916369E+01    370 6.9E+01     i

 Major Minors     Step   nCon Feasible  Optimal  MeritFunction     nS Penalty
   100      1  1.0E+00    111 (5.0E-08) 3.3E-06  4.5912602E+01    370 6.9E+01   R i
   101      2  1.0E+00    112 (5.0E-08)(1.9E-11) 4.5910581E+01    371 6.9E+01 s   i

 SNOPTC EXIT   0 -- finished successfully
 SNOPTC INFO   1 -- optimality conditions satisfied

 Problem name                 solver
 No. of iterations                 586   Objective            4.5910580616E+01
 No. of major iterations           101   Linear    obj. term  0.0000000000E+00
 Penalty parameter           6.855E+01   Nonlinear obj. term  4.5910580616E+01
 User function calls (total)       113
 No. of superbasics                371   No. of basic nonlinears           743
 No. of degenerate steps             6   Percentage                       1.02
 Max x                     431 4.3E+00   Max pi                   1697 1.0E+10
 Max Primal infeas        1134 2.2E-07   Max Dual infeas           722 1.9E-01
 Nonlinear constraint violn    2.2E-07

 Solution printed on file  10

 Time for MPS input                             0.00 seconds
 Time for solving problem                       2.20 seconds
 Time for solution output                       0.02 seconds
 Time for constraint functions                  0.07 seconds
 Time for objective function                    0.00 seconds
      solver  :   t_proc      (avg)   t_wall      (avg)    n_eval
    nlp_grad  |  95.00us ( 95.00us)  94.35us ( 94.35us)         1
   nlp_jac_f  |   3.96ms ( 34.43us)   3.89ms ( 33.87us)       115
   nlp_jac_g  |   5.75ms ( 49.97us)   5.76ms ( 50.08us)       115
       total  |   2.21 s (  2.21 s)   2.21 s (  2.21 s)         1
traversaro commented 3 years ago

This is still interesting to investigate, but at this point I would wait for the new Ipopt 3.14 to investigate this, in the meanwhile I remove myself as assignee to more easily track by immediate backlog.

traversaro commented 3 years ago

I wonder if this is related: https://github.com/coin-or/Ipopt/issues/491 .

traversaro commented 3 years ago

It would be interesting also to run the test with pardiso.

FYI a conda build with pardiso mkl is being prepared in https://github.com/conda-forge/ipopt-feedstock/pull/65 . Note that pardiso mkl may be slower then normal pardiso (see https://www.pardiso-project.org/) so effectively pardisomkl and pardiso are two different solvers, but neverthless it would be nice to test it.

traversaro commented 3 years ago

I re-run the test with latest coinbrew and latest blf and latest conda packages. Interesting, the latest coinbrew is now installing MUMPS 5.* , see https://github.com/coin-or/Ipopt/issues/479#issuecomment-850474322 . Furthermore, Ipopt has been updated to 3.14.1 in coinbrew and conda, while on apt it remains to the old version 3.11.9, see https://repology.org/project/ipopt/versions . See https://github.com/traversaro/ipopt-walking-benchmarks/runs/2983648755 . Note that is possible that also the machine instances used by GitHub Actions changed, so the only relevant thing to check are relativing timings.

The new results are:

Name Time spent executing TimeVaryingDCMPlannerUnitTests (s) Description
conda-manual 1.85 All dependencies installed via conda (conda-forge, robotology-staging), bfl compiled manually
conda-superbuild 3.5 mumps, ipopt installed via conda (conda-forge), casadi and bfl installed via superbuild
conda-coinbrew-superbuild 0.9 mumps, ipopt installed via coinbrew (conda-forge), casadi and bfl installed via superbuild with conda-forge deps
apt-superbuild 0.72 mumps, ipopt installed via apt, casadi and blf sia superbuild
apt-coinbrew-superbuild 0.5 mumps, ipopt installed via coinbrew, casadi and blf sia superbuild with apt deps
traversaro commented 3 years ago

What is really strange, even w.r.t. to previous builds, is the different betwen conda-manual and conda-superbuild. In theory the only difference there is the version of casadi (built via superbuilda vs built in conda), and yet the number of iterations and the time spent is completly different. Something is fishy.

traversaro commented 3 years ago

One difference that is particularly strange (especially considering that is not just a timing difference, but a difference in the number of the iterations of the ipopt algorithm) is the difference between conda-manual and conda-superbuild . Both this builds use exactly the same ipopt and mumps version, and also the casadi version is the same, the only difference should be that in one case casadi is taken from conda-forge binaries, and in another it is built by the superbuild. To avoid polluting this issue again, I will open a separate issue for that.

traversaro commented 3 years ago

As emerged in https://github.com/traversaro/ipopt-walking-benchmarks/issues/2, the major difference between apt and conda is that in apt the default blas implementation is netlib, while on conda is openblas on Linux/macOS and mkl on Windows. By switching to using netlib on conda as well, we finally get a consistent number of iterations (from https://github.com/traversaro/ipopt-walking-benchmarks/runs/2998363108):

conda-coinbrew-superbuild

 20: ******************************************************************************
20: This program contains Ipopt, a library for large-scale nonlinear optimization.
20:  Ipopt is released as open source code under the Eclipse Public License (EPL).
20:          For more information visit https://github.com/coin-or/Ipopt
20: ******************************************************************************
20: 
20:       solver  :   t_proc      (avg)   t_wall      (avg)    n_eval
20:        nlp_f  | 907.00us ( 18.51us) 904.47us ( 18.46us)        49
20:        nlp_g  |   1.39ms ( 28.45us)   1.37ms ( 27.89us)        49
20:     nlp_grad  |  95.00us ( 95.00us)  94.41us ( 94.41us)         1
20:   nlp_grad_f  | 740.00us ( 33.64us) 779.66us ( 35.44us)        22
20:   nlp_hess_l  |   1.30ms ( 68.32us)   1.30ms ( 68.30us)        19
20:    nlp_jac_g  |   1.47ms ( 63.74us)   1.46ms ( 63.67us)        23
20:        total  | 358.38ms (358.38ms) 358.52ms (358.52ms)         1
20: ===============================================================================
20: All tests passed (157 assertions in 1 test case)
20: 
    Test #20: TimeVaryingDCMPlannerUnitTests ...   Passed    0.50 sec
    Start 20: TimeVaryingDCMPlannerUnitTests

conda-superbuild

 20: ******************************************************************************
20: This program contains Ipopt, a library for large-scale nonlinear optimization.
20:  Ipopt is released as open source code under the Eclipse Public License (EPL).
20:          For more information visit https://github.com/coin-or/Ipopt
20: ******************************************************************************
20: 
20:       solver  :   t_proc      (avg)   t_wall      (avg)    n_eval
20:        nlp_f  | 762.00us ( 15.55us) 761.71us ( 15.55us)        49
20:        nlp_g  |   1.25ms ( 25.55us)   1.22ms ( 24.96us)        49
20:     nlp_grad  |  91.00us ( 91.00us)  91.10us ( 91.10us)         1
20:   nlp_grad_f  | 652.00us ( 29.64us) 645.81us ( 29.35us)        22
20:   nlp_hess_l  |   1.12ms ( 59.16us)   1.14ms ( 59.75us)        19
20:    nlp_jac_g  |   1.17ms ( 51.04us)   1.18ms ( 51.21us)        23
20:        total  |   1.40 s (  1.40 s)   1.37 s (  1.37 s)         1
20: ===============================================================================
20: All tests passed (157 assertions in 1 test case)
20: 
1/1 Test #20: TimeVaryingDCMPlannerUnitTests ...   Passed    1.46 sec

conda-manual

20: ******************************************************************************
20: This program contains Ipopt, a library for large-scale nonlinear optimization.
20:  Ipopt is released as open source code under the Eclipse Public License (EPL).
20:          For more information visit https://github.com/coin-or/Ipopt
20: ******************************************************************************
20: 
20:       solver  :   t_proc      (avg)   t_wall      (avg)    n_eval
20:        nlp_f  | 870.00us ( 17.76us) 871.91us ( 17.79us)        49
20:        nlp_g  |   1.41ms ( 28.71us)   1.38ms ( 28.18us)        49
20:     nlp_grad  |  99.00us ( 99.00us)  99.20us ( 99.20us)         1
20:   nlp_grad_f  | 894.00us ( 40.64us) 890.21us ( 40.46us)        22
20:   nlp_hess_l  |   1.42ms ( 74.58us)   1.42ms ( 74.66us)        19
20:    nlp_jac_g  |   1.51ms ( 65.70us)   1.51ms ( 65.65us)        23
20:        total  |   1.50 s (  1.50 s)   1.47 s (  1.47 s)         1
20: ===============================================================================
20: All tests passed (157 assertions in 1 test case)
20: 
    Test #20: TimeVaryingDCMPlannerUnitTests ...   Passed    1.57 sec

apt-coinbrew-superbuild

20: ******************************************************************************
20: This program contains Ipopt, a library for large-scale nonlinear optimization.
20:  Ipopt is released as open source code under the Eclipse Public License (EPL).
20:          For more information visit https://github.com/coin-or/Ipopt
20: ******************************************************************************
20: 
20:       solver  :   t_proc      (avg)   t_wall      (avg)    n_eval
20:        nlp_f  | 950.00us ( 19.39us) 950.01us ( 19.39us)        49
20:        nlp_g  |   1.61ms ( 32.90us)   1.58ms ( 32.22us)        49
20:     nlp_grad  | 106.00us (106.00us) 106.30us (106.30us)         1
20:   nlp_grad_f  | 917.00us ( 41.68us) 913.30us ( 41.51us)        22
20:   nlp_hess_l  |   1.45ms ( 76.21us)   1.45ms ( 76.20us)        19
20:    nlp_jac_g  |   1.63ms ( 70.83us)   1.63ms ( 70.92us)        23
20:        total  | 380.03ms (380.03ms) 380.06ms (380.06ms)         1
20: ===============================================================================
20: All tests passed (157 assertions in 1 test case)
20: 
1/1 Test #20: TimeVaryingDCMPlannerUnitTests ...   Passed    0.50 sec

apt-superbuild

20: ******************************************************************************
20: This program contains Ipopt, a library for large-scale nonlinear optimization.
20:  Ipopt is released as open source code under the Eclipse Public License (EPL).
20:          For more information visit http://projects.coin-or.org/Ipopt
20: ******************************************************************************
20: 
20:       solver  :   t_proc      (avg)   t_wall      (avg)    n_eval
20:        nlp_f  | 935.00us ( 15.85us) 919.16us ( 15.58us)        59
20:        nlp_g  |   1.57ms ( 26.58us)   1.57ms ( 26.62us)        59
20:     nlp_grad  | 102.00us (102.00us) 101.81us (101.81us)         1
20:   nlp_grad_f  | 988.00us ( 31.87us)   1.01ms ( 32.49us)        31
20:   nlp_hess_l  |   1.78ms ( 63.50us)   1.78ms ( 63.55us)        28
20:    nlp_jac_g  |   1.89ms ( 57.15us)   1.89ms ( 57.26us)        33
20:        total  | 588.70ms (588.70ms) 588.88ms (588.88ms)         1
20: ===============================================================================
20: All tests passed (157 assertions in 1 test case)
20: 
1/1 Test #20: TimeVaryingDCMPlannerUnitTests ...   Passed    0.71 sec
traversaro commented 3 years ago

Except for apt-superbuild that is using a really old version of Ipopt, all the other methods now exit in the same number of iterations (49). Furthermore, when coinbrew is used to compile mumps and ipopt, the performance are exactly the same on conda and on apt, with 0.5 s (even if the performance are worst then when mumps 4 was used).

At this point, it seems much more probable that the difference between the computation time is all in the linear solver (even because the time spent in the function evaluaton between all the case is comparable, and in any case is always around 6/7 ms). In particular, the difference between the conda mumps and the coinbrew mumps (resulting in 1.5 s vs 0.5 s) need to be investigate, but it is probably an effect of the specific variant built in conda (with/without metis, and similar options) and by the specific build of mumps used in ipopt built (both mumps-mpi and mump-seq are available in conda, but only mumps-seq is linked to ipopt).

traversaro commented 3 years ago

As a general comment on TimeVaryingDCMPlannerUnitTests performances, probably as mentioned in https://github.com/traversaro/ipopt-walking-benchmarks/issues/2 it may be worth to try to tune termination parameters:

The number of iterations vary a lot across blas implementations. It is 71 for mkl, 69/34 for openblas, 49 for netlib, 47 for blis. I am not sure about this, but I am afraid this is an indication that the default termination condition/threshold of ipopt may not be appropriate/ideal for this problem. TL;DR: If we are interested in optimizing TimeVaryingDCMPlannerUnitTests planning time, it may be worth to try to tune some terminating condition, perhaps plotting the lagrangian error w.r.t. to the iteration time before.

traversaro commented 3 years ago

@GiulioRomualdi while in the future we may investigate how to have a "fast" mumps also on conda, I think that the fact that coinbrew has now the same performance on conda and apt means that we can close the issue, what do you think? I also dropped a comment on how to install netlib-based blas in https://github.com/robotology/robotology-superbuild/issues/659 .

GiulioRomualdi commented 3 years ago

This morning I had to install Ipopt from scratch I used CoinBrew to install MUMPS and IPOPT (apt-coinbrew-superbuild) Differently from the original scenario https://github.com/dic-iit/bipedal-locomotion-framework/issues/219 we have

Scenario IPOPT version MUMPS version
Original https://github.com/dic-iit/bipedal-locomotion-framework/issues/219 3.13.4 4.10.0 Tue May 10 12:56:32 UTC 2011
30 August 2021 (Latest releases) 3.14.2 5.4.0 Tue Apr 13 15:26:30 UTC 2021

Here the result

20: ******************************************************************************
20: This program contains Ipopt, a library for large-scale nonlinear optimization.
20:  Ipopt is released as open source code under the Eclipse Public License (EPL).
20:          For more information visit https://github.com/coin-or/Ipopt
20: ******************************************************************************
20: 
20:       solver  :   t_proc      (avg)   t_wall      (avg)    n_eval
20:        nlp_f  | 918.00us ( 13.30us) 914.82us ( 13.26us)        69
20:        nlp_g  |   1.69ms ( 24.51us)   1.64ms ( 23.77us)        69
20:     nlp_grad  |  70.00us ( 70.00us)  70.57us ( 70.57us)         1
20:   nlp_grad_f  |   1.00ms ( 25.69us) 998.33us ( 25.60us)        39
20:   nlp_hess_l  |   1.88ms ( 52.31us)   1.88ms ( 52.21us)        36
20:    nlp_jac_g  |   1.87ms ( 46.80us)   1.88ms ( 46.95us)        40
20:        total  | 521.60ms (521.60ms) 521.60ms (521.60ms)         1
20: ===============================================================================
20: All tests passed (157 assertions in 1 test case)
20: 
1/1 Test #20: TimeVaryingDCMPlannerUnitTests ...   Passed    0.68 sec
traversaro commented 3 years ago

A possible way to explore the fast open source combination for this problem is to use spack. The main advantage of this package manager is that it compiles everything from source optimizing the code for the specific microarchitecture on the system, and you can specify several compile time options and even the version of dependency, so you can do:

spack install ipopt ^mumps@4.1.0
spack install ipopt ^mumps@5.4.0

to install two different version of ipopt with different dependencies. Unfortunatly I noticed just know that only 5.* version of mumps are supported in spack at the moment: https://spack.readthedocs.io/en/latest/package_list.html#mumps . See https://github.com/traversaro/robotology-superbuild-spack-experiments/ if you are curious on some other experiments.