search

jump-dev / Ipopt.jl

A Julia interface to the Ipopt nonlinear solver
https://github.com/coin-or/ipopt
Other
152 stars 58 forks source link

pardiso, ipopt, undefined symbol: sgetrf_ #177

Closed pedroborgesmelo closed 3 years ago

pedroborgesmelo commented 5 years ago

Hello Guys!

I have a working/functional build of Ipopt with MKL-Blas-Lapack and Non-Mkl-Pardiso. The cpp example that comes with Ipopt works fine if I set linear_solver=pardiso.

On the julia side, I get the problem below: Exception of type: OPTIONINVALID in file "IpAlgBuilder.cpp" at line 366: Exception message: Selected linear solver Pardiso not available. Tried to obtain Pardiso from shared library "libpardiso.so", but the following error occured: /impa/home/f/pborges/Desktop/installs/pardiso/libpardiso.so: undefined symbol: sgetrf

If I try:

m = Model(solver=IpoptSolver(linear_solver="pardiso"))
@variable(m, x >= 0)
@objective(m, :Min, x)
solve(m)

my julia version is 0.6.4.

There is this related issue here: https://github.com/JuliaOpt/Clp.jl/issues/51

This other one is also related: https://github.com/JuliaOpt/Ipopt.jl/issues/106

Any clues of what is going on?

@mlubin ?

pedroborgesmelo commented 5 years ago

Hello Guys!

My solution was:

Updating my julia version. But, just upgrading the packages was not enough. I had to really get a new julia running.

michel2323 commented 5 years ago

Hi,

I run the same setup, without MKL, just openblas. I run into the same issue with Julia 1.1.1. From reading the related Clp issue it seems there might be a fix for Ipopt.jl? What is actually the issue here?

pedroborgesmelo commented 5 years ago

Hi,

I am not sure If openblas provides espack routines.

Pedro.

Em qui, 1 de ago de 2019 12:12, Michel Schanen notifications@github.com escreveu:

Hi,

I run the same setup, without MKL, just openblas. I run into the same issue with Julia 1.1.1. From reading the related Clp issue it seems there might be a fix for Ipopt.jl?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/JuliaOpt/Ipopt.jl/issues/177?email_source=notifications&email_token=AEXMSVSZG7KGXCPNR3OUO6DQCL4OJA5CNFSM4IGBPWN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3K5MSA#issuecomment-517330504, or mute the thread https://github.com/notifications/unsubscribe-auth/AEXMSVXBYAGOGOJBPPMWG43QCL4OJANCNFSM4IGBPWNQ .

pedroborgesmelo commented 5 years ago

Lapack*

Em qui, 1 de ago de 2019 17:23, Pedro Borges pedro.borges.melo@gmail.com escreveu:

Hi,

I am not sure If openblas provides espack routines.

Pedro.

Em qui, 1 de ago de 2019 12:12, Michel Schanen notifications@github.com escreveu:

Hi,

I run the same setup, without MKL, just openblas. I run into the same issue with Julia 1.1.1. From reading the related Clp issue it seems there might be a fix for Ipopt.jl?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/JuliaOpt/Ipopt.jl/issues/177?email_source=notifications&email_token=AEXMSVSZG7KGXCPNR3OUO6DQCL4OJA5CNFSM4IGBPWN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3K5MSA#issuecomment-517330504, or mute the thread https://github.com/notifications/unsubscribe-auth/AEXMSVXBYAGOGOJBPPMWG43QCL4OJANCNFSM4IGBPWNQ .

michel2323 commented 5 years ago

Yes, sorry. I use the standard packages libblas-dev and liblapack-dev that Ubuntu provides. make test in the Ipopt repo runs fine, whereas I get this error when running Ipopt and pardiso through Julia/JuMP.

Exception of type: OPTION_INVALID in file "../../../../Ipopt/src/Algorithm/IpAlgBuilder.cpp" at line 366:
 Exception message: Selected linear solver Pardiso not available.
Tried to obtain Pardiso from shared library "libpardiso.so", but the following error occured:
/home/michel/git/Ipopt/lib/libpardiso.so: undefined symbol: sgetrf_

Edit: I do not want to use the pardiso version provided through MKL.

michel2323 commented 5 years ago

For anyone running into this issue, here is a nasty workaround:

using Libdl
liblapack=Libdl.dlopen("/usr/lib/x86_64-linux-gnu/liblapack.so",RTLD_GLOBAL)
libomp=Libdl.dlopen("/usr/lib/x86_64-linux-gnu/libomp.so",RTLD_GLOBAL)

This loads the libraries and makes the symbols available for subsequently loaded libraries, i.e. pardiso.

dpo commented 4 years ago

Is there a reliable solution to this problem? I'm absolutely unable to use Ipopt with non-MKL Pardiso.

Fisheryu1234 commented 3 years ago

Is there a reliable solution to this problem? I'm absolutely unable to use Ipopt with non-MKL Pardiso.

I'm having the same issue on Mac. No idea how to use Ipopt in JuMP with non-MKL Pardiso.

odow commented 3 years ago

This PR has instructions for how I successfully used Pardiso on Mac and Linux: https://github.com/jump-dev/Ipopt.jl/pull/236.

Fisheryu1234 commented 3 years ago

Thanks a lot for your instruction. However, I still encounter the error in Julia in Mac. The detailed steps follow

  1. First got Pardiso license from the official website, I renamed the .dylib file and put under a home folder "~/Pardiso" with the license file. Terminal outputs "libpardiso.dylib Pardiso.lic" after ls /Users/xiaoyuyu/Pardiso
  2. Then export LD_LOAD_PATH=/Users/xiaoyuyu/Pardiso; julia starts Julia (version 1.5.3)
  3. Entered the test example: 
    
using Ipopt JuMP
model = Model(optimizer_with_attributes(Ipopt.Optimizer,"tol"=>1.e-14,              
             "mu_strategy" => "adaptive","linear_solver" => "Pardiso"))
@variable(model, 0 <= x <= 2)
@variable(model, 0 <= y <= 30)
@objective(model, Max, 5x + 3 * y)
@constraint(model, con, 1x + 5y <= 3)

optimize!(model)

It reports the following message:

> Exception of type: OPTION_INVALID in file "IpAlgBuilder.cpp" at line 374:
 Exception message: Selected linear solver Pardiso not available.
Tried to obtain Pardiso from shared library "libpardiso.dylib", but the following error occured:
dlopen(libpardiso.dylib, 2): image not found
EXIT: Invalid option encountered.

I further copy the "libpardiso.dylib" into the "/Applications/Julia-1.5.app/Contents/Resources/julia/lib/Julia" folder and double checked by

using Libdl dlpath("libpardiso.dylib")


and outputs

> ERROR: could not load library "libpardiso.dylib"
dlopen(libpardiso.dylib, 1): no suitable image found.  Did find:
    /Applications/Julia-1.5.app/Contents/Resources/julia/lib/julia/libpardiso.dylib: file too short
Stacktrace:
 [1] dlopen(::String, ::UInt32; throw_error::Bool) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Libdl/src/Libdl.jl:109
 [2] dlopen at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Libdl/src/Libdl.jl:109 [inlined] (repeats 2 times)
 [3] dlpath(::String) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Libdl/src/Libdl.jl:232
 [4] top-level scope at REPL[5]:1

Why is "file too short"? I am relatively inexperienced in terminal use and I couldn't find out how to load the external path of Pardiso library (here is 
"/Users/xiaoyuyu/Pardiso") into Julia environment, or other alternative approach.

Since I exclusively use Juno as IDE, I was wondering why I can't write `export LD_LOAD_PATH="/Users/xiaoyuyu/Pardiso" ` into .../startup.jl, which delivers the following error message in REPL

> ERROR: LoadError: syntax: invalid assignment location "export LD_LOAD_PATH" around /Users/xiaoyuyu/.julia/config/startup.jl:2
Stacktrace:
 [1] top-level scope at /Users/xiaoyuyu/.julia/config/startup.jl:2
 [2] include(::Function, ::Module, ::String) at ./Base.jl:380
 [3] include at ./Base.jl:368 [inlined]
 [4] include_ifexists at ./client.jl:213 [inlined]
 [5] load_julia_startup() at ./client.jl:328
 [6] exec_options(::Base.JLOptions) at ./client.jl:267
 [7] _start() at ./client.jl:506
in expression starting at /Users/xiaoyuyu/.julia/config/startup.jl:2
Julia has exited.

I had separate issue on compling "Pardiso.jl", but my main goal is to use Parsido as the linear solver for Ipopt in JuMP.

Appreciate any help.
odow commented 3 years ago

What's the output of 

using Libdl
dlopen("/Users/xiaoyuyu/Pardiso/libpardiso.dylib")

The file too short suggests the file might be corrupt. Try downloading a fresh copy. I downloaded libpardiso600-MACOS-X86-64.dylib.

Fisheryu1234 commented 3 years ago

I first re-downloaded libpardiso600-MACOS-X86-64.dylib from the official website, renamed, and moved into /Users/xiaoyuyu/Pardiso

Then from Julia I run your command and got

Ptr{Nothing} @0x00007f8cc1514340

I believed I missed something.

odow commented 3 years ago

That looks like it worked. Check the following works. If it does, you should be able to run Ipopt.

export LD_LOAD_PATH=/Users/xiaoyuyu/Pardiso; julia -e "using Libdl; dlopen(\"libpardiso.dylib\")"
Fisheryu1234 commented 3 years ago

I still get the following error message from above command:

ERROR: could not load library "libpardiso.dylib" dlopen(libpardiso.dylib, 1): image not found Stacktrace: [1] dlopen(::String, ::UInt32; throw_error::Bool) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Libdl/src/Libdl.jl:109 [2] dlopen at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Libdl/src/Libdl.jl:109 [inlined] (repeats 2 times) [3] top-level scope at none:1

I guess it is because I failed the path loading even starting with export LD_LOAD_PATH=/Users/xiaoyuyu/Pardiso. However, when I check with println(dllist(). I find /Users/xiaoyuyu/Pardiso/libpardiso.dylib once out of three times. The output from dllist() seems random each time.

odow commented 3 years ago

The output from dllist() seems random each time.

Something seems very strange with your machine. If you try solving the Ipopt model does it find Pardiso once out of every three time as well?

Fisheryu1234 commented 3 years ago

No, the Ipopt fails to find Pardiso all the time. I am giving up now.

odow commented 3 years ago

According to this issue, Pardiso hard-codes some paths on Mac: https://github.com/JuliaSparse/Pardiso.jl/issues/29#issuecomment-361531593 Do you have those files? That might explain the weird behavior.

Edit: although I just checked, and I don't have those.

odow commented 3 years ago

@Fisheryu1234 I've led you down the wrong path with a horrible typo: It should be DL_LOAD_PATH, not LD_LOAD_PATH.

odow commented 3 years ago

Closing in favor of #247