BUG: Tried to obtain HSL_MA86 from shared library "libhsl.dylib"

[ermalrrapaj]

Hi, I'm new to Julia so might need some help. I have already installed Ipopt and linked it to HSL libraries on my Ubuntu. I have also tested that it works in c++/fortran and also in python. However, the Ipopt wrapper for Julia has been giving me a headache as it never works. I followed the instructions:

ENV["JULIA_IPOPT_LIBRARY_PATH"] = "/usr/lib"
ENV["JULIA_IPOPT_EXECUTABLE_PATH"] = "/usr/bin"
import Pkg; Pkg.build("Ipopt")

and it fails:

RROR: The following package names could not be resolved:
 * Ipopt (not found in project or manifest)

Stacktrace:
 [1] pkgerror(::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/Types.jl:53
 [2] ensure_resolved(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; registry::Bool) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/Types.jl:802
 [3] ensure_resolved at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/Types.jl:767 [inlined]
 [4] build(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; verbose::Bool, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:697
 [5] build at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:682 [inlined]
 [6] #build#108 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:680 [inlined]
 [7] build at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:680 [inlined]
 [8] #build#105 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:677 [inlined]
 [9] build(::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:677
 [10] top-level scope at REPL[4]:1

Then, I try to install the default one, but I set up the path so that it links properly:

ENV["JULIA_IPOPT_LIBRARY_PATH"] = "/usr/lib"
ENV["JULIA_IPOPT_EXECUTABLE_PATH"] = "/usr/bin"
import Pkg; Pkg.add("Ipopt")

It installs but fails to find the hsl libraries with the following error:

Exception of type: OPTION_INVALID in file "IpAlgBuilder.cpp" at line 268:
 Exception message: Selected linear solver MA27 not available.
Tried to obtain MA27 from shared library "libhsl.so", but the following error occured:
libhsl.so: cannot open shared object file: No such file or directory

Do you know if it's possible to use HSL with Ipopt in Julia? If yes, can you please tell me how to proceed? I followed the instructions provided but nothing works so far. Thank you.

[odow]

When you installed HSL where did you install it? i.e., where is libhls.so?

Edit: ignore my initial comment, it looks like you did try to follow https://github.com/JuliaOpt/Ipopt.jl/issues/189.

[ermalrrapaj]

Hi, I have version 3.13.12 of Ipopt installed and in #189 they had a different version. So, I followed the instructions from the Coin-Or project and did the installation with coinbrew and put coinhsl into ThirdParty/HSL just like the instructions say. I did test that it all works in multiple languages as already mentioned so my guess is that it might be a linking issues or setting up properly the path for Ipopt in julia to find the libraries. I have a local folder where I have both ipopt and hsl, mumps etc installed:

home/ermal/workspace/dist/lib/libcoinasl.la        
/home/ermal/workspace/dist/lib/libcoinmumps.la                  /home/ermal/workspace/dist/lib/libipopt.la
/home/ermal/workspace/dist/lib/libcoinasl.so        
/home/ermal/workspace/dist/lib/libcoinmumps.so                  /home/ermal/workspace/dist/lib/libipopt.so
/home/ermal/workspace/dist/lib/libcoinasl.so.2     
 /home/ermal/workspace/dist/lib/libcoinmumps.so.2                /home/ermal/workspace/dist/lib/libipopt.so.3
/home/ermal/workspace/dist/lib/libcoinasl.so.2.0.0  /home/ermal/workspace/dist/lib/libcoinmumps.so.2.1.0            /home/ermal/workspace/dist/lib/libipopt.so.3.13.2
/home/ermal/workspace/dist/lib/libcoinhsl.la        /home/ermal/workspace/dist/lib/libipoptamplinterface.la         /home/ermal/workspace/dist/lib/libsipopt.la
/home/ermal/workspace/dist/lib/libcoinhsl.so        /home/ermal/workspace/dist/lib/libipoptamplinterface.so         /home/ermal/workspace/dist/lib/libsipopt.so
/home/ermal/workspace/dist/lib/libcoinhsl.so.2      /home/ermal/workspace/dist/lib/libipoptamplinterface.so.3       /home/ermal/workspace/dist/lib/libsipopt.so.3
/home/ermal/workspace/dist/lib/libcoinhsl.so.2.1.0  /home/ermal/workspace/dist/lib/libipoptamplinterface.so.3.13.2  /home/ermal/workspace/dist/lib/libsipopt.so.3.13.2

and also:

/home/ermal/workspace/dist/lib/pkgconfig:
coinasl.pc  coinhsl.pc  coinmumps.pc  ipoptamplinterface.pc  ipopt.pc

and the bin:

/home/ermal/workspace/dist/bin/ipopt  /home/ermal/workspace/dist/bin/ipopt_sens

At first I tried to set the environment variables to this folder:

ENV["JULIA_IPOPT_LIBRARY_PATH"] = "/home/ermal/workspace/dist/lib"
ENV["JULIA_IPOPT_EXECUTABLE_PATH"] = "/home/ermal/workspace/dist/bin"

and I got the same error reported. Then I did an installation in urs/local/ but the same error. Please note that I have already set up the path in my ~/.bashrc so that the ..wrokspace/dist is included. Sorry for such a long reply but I did try. I suspect it might be a naming issue, cause I have libcoinhsl.so and not libhsl.so Should I change something in the package files in .julia perhaps?

[ermalrrapaj]

I meant /usr rather than /usr/local

[blegat]

If you get the error Ipopt (not found in project or manifest), it simply means that you need to do Pkg.add("Ipopt")

[ermalrrapaj]

Hi, Please read the full original comment I posted before replaying because I have already tried that ad reported that it doesn't solve my issues.

The installation works with Pkg.add but it can not find hsl libraries which is why I posted a comment in the first place. I'm sure it's a simple problem to solve and sorry for taking your time, but I have followed everything to the letter in both your instructions and by looking at other posts here.

How about, in your reply, you provide a step by step guide on how to install ipopt.jl based on a custom installation of ipopt already present in a linux system and show how to link to hsl libraries if one has them already compiled? Thank you.

[odow]

It seems like Ipopt is looking for libhsl.so, but it has been installed as libcoinhsl.so. I don't have time to dig into why this is happening, so here is what I did to get MA27 working on Mac just now:

1. Download HSL from http://www.hsl.rl.ac.uk/ipopt/
2. Unzip it, ./configure, make install
3. Rename /usr/local/lib/libcoinhsl.dylib to /usr/local/lib/libhsl.dylib

4. 
   using Ipopt, JuMP
   model = Model(() -> Ipopt.Optimizer(linear_solver = "ma27"))
   @variable(model, 0 <= x <= 2)
   @variable(model, 0 <= y <= 30)
   @objective(model, Max, 5x + 3 * y)
   @constraint(model, con, 1x + 5y <= 3)

julia> optimize!(model)

---

This program contains Ipopt, a library for large-scale nonlinear optimization. Ipopt is released as open source code under the Eclipse Public License (EPL). For more information visit http://projects.coin-or.org/Ipopt

---

This is Ipopt version 3.13.2, running with linear solver ma27.

Number of nonzeros in equality constraint Jacobian...: 0 Number of nonzeros in inequality constraint Jacobian.: 2 Number of nonzeros in Lagrangian Hessian.............: 0

Note that you don't need to specially compile Ipopt; just use the provided binaries.

> How about, in your reply, you provide a step by step guide on how to install ipopt.jl based on a custom installation of ipopt already present in a linux system and show how to link to hsl libraries if one has them already compiled?

Please remember that Ipopt.jl is provided and supported by volunteers.

[ermalrrapaj]

Hi, First of all, sorry if my previous comment came across as ungrateful (or rude). I appreciate all the help given, and wouldn't mind contributing once I have more experience. I actually was thinking I was making some stupid mistake in my installation or missing a step somewhere, so having all the steps listed would make sure I didn't mess it up. Anyways, thanks a lot for the last reply as it helped me solve my issues. I just did a tiny modification for ubuntu (I get .so files) and it worked. I am posting here a list of all steps needed for anyone who happens to face the same issue: (1) install Ipopt.l: import Pkg Pkg.add("Ipopt") (2) download and install hsl following the instructions (steps 1 and 2) above from Oscar. (3) depending on the OS it might be .so or .dll file. In my case I had libcoinhsl.la and libcoinhsl.so libcoinhsl.so.2 libcoinhsl.so.2.1.0. I had to copy them to /usr/lib and then rename them to libhsl.la libhsl.so libhsl.so.2 and libhsl.so.2.1.0 I also made sure the names inside libhsl.la matched the new names. (4) it should all work out just fine from here. In my case putting the libraries in usr/local/lib does not resolve the problem but in /usr/lib it does. I guess that depending on the OS, Ipopt.jl is looking for the libraries in different paths. (*) I did try with libcoinhs.la etc at first and it didn't work. So these files have to be renamed. () If I may suggest, it would be nice to have some ENV variable for hsl, paradiso etc and then do Pkg.build("Ipopt"). I guess it would resolve all these issues of finding the libraries and it would be the same procedure for mac, windows and linux. I have no idea how to do this, but hopefully someone here can.
Thanks again, and I'm closing this thread since it's resolved.

[odow]

I'm going to leave this open so that people see it in future. We definitely need to improve the documentation and ease-of-use for Ipopt and HSL.

[odow]

It appears that Ipopt has been compiled to search for libhsl.xx, but HSL now installs libcoinhsl.xx.

The action item here is to figure out if this is a due to a recent change in HSL, coin-or/ipopt, or Ipopt_jll and upstream the required changes.

[dpo]

Ipopt_jll doesn't do anything about HSL. I'm working on MKL/Pardiso though...

[odow]

I wondered if there were some compiler flags that we changed going from binary builder to Ipopt_jll. I haven't looked in any detail though. It seems more likely that HSL renamed their file?

[MartaVanin]

ignore my previous comment, I have realized it was addressed in #212 already, sorry

[ksun46]

Hi all. I encounter the same issue recently. I use Mac and have a compiled version of IPOPT with HSL library installed under my home directory. I am starting a new project, and after adding the latest version of JuMP (v0.21.5) and Ipopt(v0.6.3), I did

ENV["JULIA_IPOPT_LIBRARY_PATH"] = "/Users/ks/CoinOR/Ipopt/build/lib"
ENV["JULIA_IPOPT_EXECUTABLE_PATH"] =  "/Users/ks/CoinOR/Ipopt/build/bin"
Pkg.build("Ipopt")

and then run a toy example

m_test = Model(with_optimizer(Ipopt.Optimizer, linear_solver = "ma27"))
@variable(m_test, x[i in 1:10], lower_bound = 0.0, upper_bound = 1.0)
@constraint(m_test, sum(x) == 1.0)
@objective(m_test, Min, sum(randn() * x[i] for i in 1:10))
JuMP.optimize!(m_test)

I got the same error message

Exception of type: OPTION_INVALID in file "IpAlgBuilder.cpp" at line 268:
 Exception message: Selected linear solver MA27 not available.
Tried to obtain MA27 from shared library "libhsl.dylib", but the following error occured:
dlopen(libhsl.dylib, 2): image not found

EXIT: Invalid option encountered.

I tried to rename /Users/ks/CoinOR/Ipopt/build/lib/libcoinhsl.dylib to /Users/ks/CoinOR/Ipopt/build/lib/libhsl.dylib, but didn't solve the problem. I don't have any libcoinhsl.dylib under my root dir /usr/lib or /usr/local/lib.

My installation of Ipopt/HSL should be fine since I can call MA27/MA57 from C++ programs. Also, the above procedure used to work for me in my previous Julia projects (with older JuMP.jl/Ipopt.jl) on the same machine. Any suggestions? Thanks.

[odow]

236 added better instructions for how to use custom installations of Ipopt (https://github.com/jump-dev/Ipopt.jl#custom-installation). It also describes how you can use MA27 on Mac (https://github.com/jump-dev/Ipopt.jl#mac-2).

I'm going to close this issue in favor of #247. If the instructions are still unclear, please post there.