search

mechmotum / cyipopt

Cython interface for the interior point optimzer IPOPT
Eclipse Public License 2.0
227 stars 54 forks source link

is it possible to include ma27 and spral in conda? #233

Closed edwardsun closed 11 months ago

edwardsun commented 11 months ago

Hi,

I'm wondering if it is possible to include free solvers such as MA27 and SPRAL in the conda installer. It seems current cyipopt uses the default MUMPS as the linear solver. Accoriding to https://or.stackexchange.com/questions/2669/ipopt-with-hsl-vs-mumps, MA27 and SPRAL are also free to use. However, the installation seems a bit complicated. Ideally it would be great to do a simple pip install or conda install.

Hence the request. Thanks for your consideration.

moorepants commented 11 months ago

Spral has been included with the conda-forge ipopt as of a month or two ago (but only on linux); https://github.com/conda-forge/ipopt-feedstock/pull/94

You can open an issue there for MA27 but my guess is that the license is not compatible with distribution via conda forge.

Here are instructions for using the HSL libraries with cyipopt: https://cyipopt.readthedocs.io/en/stable/install.html#conda-forge-binaries-with-hsl

edwardsun commented 11 months ago

Thank you. Has the change been effective in conda? In a linux docker container, I did: # conda list | grep "cyipopt" cyipopt 1.3.0 py310hd3d3a20_0 conda-forge

Yet I still get:

Setting: "spral" is not a valid setting for Option: linear_solver. Check the option documentation.

linear_solver (String)

Category: Linear Solver Description: Linear solver used for step computations. Valid Settings: ma27 (load the Harwell routine MA27 from library at runtime) ma57 (load the Harwell routine MA57 from library at runtime) ma77 (load the Harwell routine HSL_MA77 from library at runtime) ma86 (load the Harwell routine MA86 from library at runtime) ma97 (load the Harwell routine MA97 from library at runtime) pardiso (load the Pardiso package from pardiso-project.org from user-provided library at runtime) mumps (use the Mumps package) custom (use custom linear solver (expert use)) Default: "mumps" Traceback (most recent call last):

Thank you for your help!

moorepants commented 11 months ago

I don't know. This isn't a cyipopt issue. This is a ipopt issue. If you have trouble with the conda forge ipopt, then I recommend opening an issue on that feedstock.

edwardsun commented 11 months ago

I don't know. This isn't a cyipopt issue. This is a ipopt issue. If you have trouble with the conda forge ipopt, then I recommend opening an issue on that feedstock.

Hi, I'm confused. What's the connection between https://github.com/conda-forge/ipopt-feedstock and cyipopt? I'm using cyipopt in conda. Thanks.

moorepants commented 11 months ago

cyipopt is a python wrapper to ipopt. Both programs are available via conda forge. The "feedstocks" are the github repositories that are responsible for the build and packaging of software on conda forge. The ipopt-feedstock and the spral-feedstock are where all information is that controls whether spral can be used with the conda forge ipopt. This repository (cyipopt) and the conda forge (cyipopt-feedstock) do not have any thing to do with how ipopt is built for conda forge (e.g whether spral is included or not). cyipopt should support anything that the ipopt it is built against supports. So if you want to know why spral isn't accessible from cyipopt, then you have to look at the ipopt that cyipopt is built against to see if it actually includes spral or if there is some other issue. In the case of conda-forge, that is the ipopt-feedstock.

moorepants commented 11 months ago

spral works as expected for me on Linux:

(cyipopt-dev) moorepants@nandi:cyipopt(master)$ conda list | grep spral
libspral                  2023.08.02           h2baf039_0    conda-forge
(cyipopt-dev) moorepants@nandi:cyipopt(master)$ conda list | grep ipopt
# packages in environment at /home/moorepants/miniconda/envs/cyipopt-dev:
cyipopt                   1.4.0.dev0                dev_0    <develop>
ipopt                     3.14.12              he6d3896_1    conda-forge
(cyipopt-dev) moorepants@nandi:cyipopt(master)$ git diff
diff --git a/examples/hs071.py b/examples/hs071.py
index d2661eb..f55cc0e 100644
--- a/examples/hs071.py
+++ b/examples/hs071.py
@@ -136,6 +136,7 @@ def main():
     #nlp.addOption('derivative_test', 'second-order')
     nlp.add_option('mu_strategy', 'adaptive')
     nlp.add_option('tol', 1e-7)
+    nlp.add_option('linear_solver', 'spral')

     #
     # Scale the problem (Just for demonstration purposes)
(cyipopt-dev) moorepants@nandi:cyipopt(master)$ python examples/hs071.py 

******************************************************************************
This program contains Ipopt, a library for large-scale nonlinear optimization.
 Ipopt is released as open source code under the Eclipse Public License (EPL).
         For more information visit https://github.com/coin-or/Ipopt
******************************************************************************

This is Ipopt version 3.14.12, running with linear solver spral.

Number of nonzeros in equality constraint Jacobian...:        4
Number of nonzeros in inequality constraint Jacobian.:        4
Number of nonzeros in Lagrangian Hessian.............:       10

In SpralSolverInterface::Factorization: Unhandled error. info.flag = -53.
Maybe one forgot to set environment variable OMP_CANCELLATION to TRUE.
Set spral_print_level=0 to see more details.
Total number of variables............................:        4
                     variables with only lower bounds:        0
                variables with lower and upper bounds:        4
                     variables with only upper bounds:        0
Total number of equality constraints.................:        1
Total number of inequality constraints...............:        1
        inequality constraints with only lower bounds:        1
   inequality constraints with lower and upper bounds:        0
        inequality constraints with only upper bounds:        0

Objective value at iteration #0 is - 16.1097
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
   0  1.6109693e+01 1.12e+01 2.41e+01   0.0 0.00e+00    -  0.00e+00 0.00e+00   0
In SpralSolverInterface::Factorization: Unhandled error. info.flag = -53.
Maybe one forgot to set environment variable OMP_CANCELLATION to TRUE.
Set spral_print_level=0 to see more details.
WARNING: Problem in step computation; switching to emergency mode.
Objective value at iteration #1 is - 16.1097
   1r 1.6109693e+01 1.12e+01 9.99e+02   1.1 0.00e+00    -  0.00e+00 0.00e+00R  1
In SpralSolverInterface::Factorization: Unhandled error. info.flag = -53.
Maybe one forgot to set environment variable OMP_CANCELLATION to TRUE.
Set spral_print_level=0 to see more details.
WARNING: Problem in step computation; switching to emergency mode.
Cannot call restoration phase at point that is almost feasible for the restoration NLP (violation 0.000000e+00).
Abort in line search due to no other fall back.
Step computation in the restoration phase failed.

Number of Iterations....: 1

                                   (scaled)                 (unscaled)
Objective...............:   3.2219385838397756e+01    1.6109692919198878e+01
Dual infeasibility......:   2.4118799920375995e+01    1.2059399960187998e+01
Constraint violation....:   1.1243400939696002e+01    1.1243400939696002e+01
Variable bound violation:   0.0000000000000000e+00    0.0000000000000000e+00
Complementarity.........:   3.9900000599999998e+00    1.9950000299999999e+00
Overall NLP error.......:   2.4118799920375995e+01    1.2059399960187998e+01

Number of objective function evaluations             = 2
Number of objective gradient evaluations             = 2
Number of equality constraint evaluations            = 2
Number of inequality constraint evaluations          = 2
Number of equality constraint Jacobian evaluations   = 2
Number of inequality constraint Jacobian evaluations = 2
Number of Lagrangian Hessian evaluations             = 2
Total seconds in IPOPT                               = 1.724

EXIT: Restoration Failed!
Solution of the primal variables: x=array([1.00999999, 4.96000005, 4.96000005, 1.00999999])

Solution of the dual variables: lambda=array([0., 0.])

Objective=16.109692919198878
moorepants commented 11 months ago

Except that the restoration failed, just noticed that.

edwardsun commented 11 months ago

spral works as expected for me on Linux:

(cyipopt-dev) moorepants@nandi:cyipopt(master)$ conda list | grep spral
libspral                  2023.08.02           h2baf039_0    conda-forge
(cyipopt-dev) moorepants@nandi:cyipopt(master)$ conda list | grep ipopt
# packages in environment at /home/moorepants/miniconda/envs/cyipopt-dev:
cyipopt                   1.4.0.dev0                dev_0    <develop>
ipopt                     3.14.12              he6d3896_1    conda-forge
(cyipopt-dev) moorepants@nandi:cyipopt(master)$ git diff
diff --git a/examples/hs071.py b/examples/hs071.py
index d2661eb..f55cc0e 100644
--- a/examples/hs071.py
+++ b/examples/hs071.py
@@ -136,6 +136,7 @@ def main():
     #nlp.addOption('derivative_test', 'second-order')
     nlp.add_option('mu_strategy', 'adaptive')
     nlp.add_option('tol', 1e-7)
+    nlp.add_option('linear_solver', 'spral')

     #
     # Scale the problem (Just for demonstration purposes)
(cyipopt-dev) moorepants@nandi:cyipopt(master)$ python examples/hs071.py 

******************************************************************************
This program contains Ipopt, a library for large-scale nonlinear optimization.
 Ipopt is released as open source code under the Eclipse Public License (EPL).
         For more information visit https://github.com/coin-or/Ipopt
******************************************************************************

This is Ipopt version 3.14.12, running with linear solver spral.

Number of nonzeros in equality constraint Jacobian...:        4
Number of nonzeros in inequality constraint Jacobian.:        4
Number of nonzeros in Lagrangian Hessian.............:       10

In SpralSolverInterface::Factorization: Unhandled error. info.flag = -53.
Maybe one forgot to set environment variable OMP_CANCELLATION to TRUE.
Set spral_print_level=0 to see more details.
Total number of variables............................:        4
                     variables with only lower bounds:        0
                variables with lower and upper bounds:        4
                     variables with only upper bounds:        0
Total number of equality constraints.................:        1
Total number of inequality constraints...............:        1
        inequality constraints with only lower bounds:        1
   inequality constraints with lower and upper bounds:        0
        inequality constraints with only upper bounds:        0

Objective value at iteration #0 is - 16.1097
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
   0  1.6109693e+01 1.12e+01 2.41e+01   0.0 0.00e+00    -  0.00e+00 0.00e+00   0
In SpralSolverInterface::Factorization: Unhandled error. info.flag = -53.
Maybe one forgot to set environment variable OMP_CANCELLATION to TRUE.
Set spral_print_level=0 to see more details.
WARNING: Problem in step computation; switching to emergency mode.
Objective value at iteration #1 is - 16.1097
   1r 1.6109693e+01 1.12e+01 9.99e+02   1.1 0.00e+00    -  0.00e+00 0.00e+00R  1
In SpralSolverInterface::Factorization: Unhandled error. info.flag = -53.
Maybe one forgot to set environment variable OMP_CANCELLATION to TRUE.
Set spral_print_level=0 to see more details.
WARNING: Problem in step computation; switching to emergency mode.
Cannot call restoration phase at point that is almost feasible for the restoration NLP (violation 0.000000e+00).
Abort in line search due to no other fall back.
Step computation in the restoration phase failed.

Number of Iterations....: 1

                                   (scaled)                 (unscaled)
Objective...............:   3.2219385838397756e+01    1.6109692919198878e+01
Dual infeasibility......:   2.4118799920375995e+01    1.2059399960187998e+01
Constraint violation....:   1.1243400939696002e+01    1.1243400939696002e+01
Variable bound violation:   0.0000000000000000e+00    0.0000000000000000e+00
Complementarity.........:   3.9900000599999998e+00    1.9950000299999999e+00
Overall NLP error.......:   2.4118799920375995e+01    1.2059399960187998e+01

Number of objective function evaluations             = 2
Number of objective gradient evaluations             = 2
Number of equality constraint evaluations            = 2
Number of inequality constraint evaluations          = 2
Number of equality constraint Jacobian evaluations   = 2
Number of inequality constraint Jacobian evaluations = 2
Number of Lagrangian Hessian evaluations             = 2
Total seconds in IPOPT                               = 1.724

EXIT: Restoration Failed!
Solution of the primal variables: x=array([1.00999999, 4.96000005, 4.96000005, 1.00999999])

Solution of the dual variables: lambda=array([0., 0.])

Objective=16.109692919198878

I think cyipopt's latest version on conda is still 1.3.0 and it does not support spral yet...

I started a fresh conda container and tried cyipopt as follows:

  1. install cyipopt: conda install -c conda-forge cyipopt -y
  2. check versions: (base) root@339e9b77cc7d:/# conda list | grep spral (base) root@339e9b77cc7d:/# conda list | grep ipopt cyipopt 1.3.0 py310hd3d3a20_0 conda-forge ipopt 3.14.12 hf9e1ecf_0 conda-forge
  3. add linear_solver "spral" to example hs071.py and run: (base) root@339e9b77cc7d:/# python3 /download/hs071.py Setting: "spral" is not a valid setting for Option: linear_solver. Check the option documentation.

linear_solver (String)

Category: Linear Solver Description: Linear solver used for step computations. Valid Settings: ma27 (load the Harwell routine MA27 from library at runtime) ma57 (load the Harwell routine MA57 from library at runtime) ma77 (load the Harwell routine HSL_MA77 from library at runtime) ma86 (load the Harwell routine MA86 from library at runtime) ma97 (load the Harwell routine MA97 from library at runtime) pardiso (load the Pardiso package from pardiso-project.org from user-provided library at runtime) mumps (use the Mumps package) custom (use custom linear solver (expert use)) Default: "mumps" Traceback (most recent call last): File "/download/hs071.py", line 163, in main() File "/download/hs071.py", line 148, in main nlp.add_option('linear_solver','spral') File "cyipopt/cython/ipopt_wrapper.pyx", line 506, in ipopt_wrapper.Problem.add_option TypeError: Error while assigning an option

moorepants commented 11 months ago

I think cyipopt's latest version on conda is still 1.3.0 and it does not support spral yet...

I just showed you that it does work with Spral. For some reason you are not installing Spral.

moorepants commented 11 months ago

Once again, I do not think this is an issue cyipopt. I recommend asking for help with installation issues on the appropriate conda forge feedstock.

moorepants commented 11 months ago

Here is a new conda env that works with spral:

$ conda create -n cyipopt-spral ipopt cyipopt libspral
Channels:
 - conda-forge
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/moorepants/miniconda/envs/cyipopt-spral

  added / updated specs:
    - cyipopt
    - ipopt
    - libspral

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    _libgcc_mutex-0.1          |      conda_forge           3 KB  conda-forge
    _openmp_mutex-4.5          |            2_gnu          23 KB  conda-forge
    cyipopt-1.3.0              |  py311he86572f_0         186 KB  conda-forge
    libblas-3.9.0              |18_linux64_openblas          14 KB  conda-forge
    libcblas-3.9.0             |18_linux64_openblas          14 KB  conda-forge
    libgfortran-ng-13.2.0      |       h69a702a_2          23 KB  conda-forge
    liblapack-3.9.0            |18_linux64_openblas          14 KB  conda-forge
    python-3.11.6              |hab00c5b_0_cpython        29.3 MB  conda-forge
    python_abi-3.11            |          4_cp311           6 KB  conda-forge
    ------------------------------------------------------------
                                           Total:        29.6 MB

The following NEW packages will be INSTALLED:

  _libgcc_mutex      conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge 
  _openmp_mutex      conda-forge/linux-64::_openmp_mutex-4.5-2_gnu 
  ampl-mp            conda-forge/linux-64::ampl-mp-3.1.0-h2cc385e_1006 
  bzip2              conda-forge/linux-64::bzip2-1.0.8-h7f98852_4 
  ca-certificates    conda-forge/linux-64::ca-certificates-2023.7.22-hbcca054_0 
  cyipopt            conda-forge/linux-64::cyipopt-1.3.0-py311he86572f_0 
  icu                conda-forge/linux-64::icu-73.2-h59595ed_0 
  ipopt              conda-forge/linux-64::ipopt-3.14.12-he6d3896_1 
  ld_impl_linux-64   conda-forge/linux-64::ld_impl_linux-64-2.40-h41732ed_0 
  libblas            conda-forge/linux-64::libblas-3.9.0-18_linux64_openblas 
  libcblas           conda-forge/linux-64::libcblas-3.9.0-18_linux64_openblas 
  libedit            conda-forge/linux-64::libedit-3.1.20191231-he28a2e2_2 
  libexpat           conda-forge/linux-64::libexpat-2.5.0-hcb278e6_1 
  libffi             conda-forge/linux-64::libffi-3.4.2-h7f98852_5 
  libgcc-ng          conda-forge/linux-64::libgcc-ng-13.2.0-h807b86a_2 
  libgfortran-ng     conda-forge/linux-64::libgfortran-ng-13.2.0-h69a702a_2 
  libgfortran5       conda-forge/linux-64::libgfortran5-13.2.0-ha4646dd_2 
  libgomp            conda-forge/linux-64::libgomp-13.2.0-h807b86a_2 
  libhwloc           conda-forge/linux-64::libhwloc-2.9.1-nocuda_h7313eea_6 
  libiconv           conda-forge/linux-64::libiconv-1.17-h166bdaf_0 
  liblapack          conda-forge/linux-64::liblapack-3.9.0-18_linux64_openblas 
  libnsl             conda-forge/linux-64::libnsl-2.0.0-hd590300_1 
  libopenblas        conda-forge/linux-64::libopenblas-0.3.24-pthreads_h413a1c8_0 
  libspral           conda-forge/linux-64::libspral-2023.08.02-h2baf039_0 
  libsqlite          conda-forge/linux-64::libsqlite-3.43.0-h2797004_0 
  libstdcxx-ng       conda-forge/linux-64::libstdcxx-ng-13.2.0-h7e041cc_2 
  libuuid            conda-forge/linux-64::libuuid-2.38.1-h0b41bf4_0 
  libxml2            conda-forge/linux-64::libxml2-2.11.5-h232c23b_1 
  libzlib            conda-forge/linux-64::libzlib-1.2.13-hd590300_5 
  metis              conda-forge/linux-64::metis-5.1.0-h59595ed_1007 
  mumps-include      conda-forge/linux-64::mumps-include-5.2.1-ha770c72_11 
  mumps-seq          conda-forge/linux-64::mumps-seq-5.2.1-h2104b81_11 
  ncurses            conda-forge/linux-64::ncurses-6.4-hcb278e6_0 
  numpy              conda-forge/linux-64::numpy-1.26.0-py311h64a7726_0 
  openssl            conda-forge/linux-64::openssl-3.1.3-hd590300_0 
  pip                conda-forge/noarch::pip-23.2.1-pyhd8ed1ab_0 
  python             conda-forge/linux-64::python-3.11.6-hab00c5b_0_cpython 
  python_abi         conda-forge/linux-64::python_abi-3.11-4_cp311 
  readline           conda-forge/linux-64::readline-8.2-h8228510_1 
  scotch             conda-forge/linux-64::scotch-6.0.9-hb2e6521_2 
  setuptools         conda-forge/noarch::setuptools-68.2.2-pyhd8ed1ab_0 
  tk                 conda-forge/linux-64::tk-8.6.13-h2797004_0 
  tzdata             conda-forge/noarch::tzdata-2023c-h71feb2d_0 
  unixodbc           conda-forge/linux-64::unixodbc-2.3.12-h661eb56_0 
  wheel              conda-forge/noarch::wheel-0.41.2-pyhd8ed1ab_0 
  xz                 conda-forge/linux-64::xz-5.2.6-h166bdaf_0 
  zlib               conda-forge/linux-64::zlib-1.2.13-hd590300_5 

Downloading and Extracting Packages:

Preparing transaction: done                                                     
Verifying transaction: done                                                     
Executing transaction: done                                                     
#                                                                               
# To activate this environment, use                                             
#                                                                               
#     $ conda activate cyipopt-spral                                            
#                                                                               
# To deactivate an active environment, use
#
#     $ conda deactivate

$ act cyipopt-spral 
(cyipopt-spral) $ cd src/cyipopt/examples/
(cyipopt-spral) $ python hs071.py 

******************************************************************************
This program contains Ipopt, a library for large-scale nonlinear optimization.
 Ipopt is released as open source code under the Eclipse Public License (EPL).
         For more information visit https://github.com/coin-or/Ipopt
******************************************************************************

This is Ipopt version 3.14.12, running with linear solver spral.

Number of nonzeros in equality constraint Jacobian...:        4
Number of nonzeros in inequality constraint Jacobian.:        4
Number of nonzeros in Lagrangian Hessian.............:       10

In SpralSolverInterface::Factorization: Unhandled error. info.flag = -53.
Maybe one forgot to set environment variable OMP_CANCELLATION to TRUE.
Set spral_print_level=0 to see more details.
Total number of variables............................:        4
                     variables with only lower bounds:        0
                variables with lower and upper bounds:        4
                     variables with only upper bounds:        0
Total number of equality constraints.................:        1
Total number of inequality constraints...............:        1
        inequality constraints with only lower bounds:        1
   inequality constraints with lower and upper bounds:        0
        inequality constraints with only upper bounds:        0

Objective value at iteration #0 is - 16.1097
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
   0  1.6109693e+01 1.12e+01 2.41e+01   0.0 0.00e+00    -  0.00e+00 0.00e+00   0
In SpralSolverInterface::Factorization: Unhandled error. info.flag = -53.
Maybe one forgot to set environment variable OMP_CANCELLATION to TRUE.
Set spral_print_level=0 to see more details.
WARNING: Problem in step computation; switching to emergency mode.
Objective value at iteration #1 is - 16.1097
   1r 1.6109693e+01 1.12e+01 9.99e+02   1.1 0.00e+00    -  0.00e+00 0.00e+00R  1
In SpralSolverInterface::Factorization: Unhandled error. info.flag = -53.
Maybe one forgot to set environment variable OMP_CANCELLATION to TRUE.
Set spral_print_level=0 to see more details.
WARNING: Problem in step computation; switching to emergency mode.
Cannot call restoration phase at point that is almost feasible for the restoration NLP (violation 0.000000e+00).
Abort in line search due to no other fall back.
Step computation in the restoration phase failed.

Number of Iterations....: 1

                                   (scaled)                 (unscaled)
Objective...............:   3.2219385838397756e+01    1.6109692919198878e+01
Dual infeasibility......:   2.4118799920375995e+01    1.2059399960187998e+01
Constraint violation....:   1.1243400939696002e+01    1.1243400939696002e+01
Variable bound violation:   0.0000000000000000e+00    0.0000000000000000e+00
Complementarity.........:   3.9900000599999998e+00    1.9950000299999999e+00
Overall NLP error.......:   2.4118799920375995e+01    1.2059399960187998e+01

Number of objective function evaluations             = 2
Number of objective gradient evaluations             = 2
Number of equality constraint evaluations            = 2
Number of inequality constraint evaluations          = 2
Number of equality constraint Jacobian evaluations   = 2
Number of inequality constraint Jacobian evaluations = 2
Number of Lagrangian Hessian evaluations             = 2
Total seconds in IPOPT                               = 0.139

EXIT: Restoration Failed!
Solution of the primal variables: x=array([1.00999999, 4.96000005, 4.96000005, 1.00999999])

Solution of the dual variables: lambda=array([0., 0.])

Objective=16.109692919198878
edwardsun commented 11 months ago

Thank you so much for helping out. I can get spral working in cyipopt 1.3.0. Meanwhile, maybe it is a good idea to specify in readme about the libspral installation, as spral significantly outperforms mumps in my case. Thanks again.