search

JuliaLinearAlgebra / Arpack.jl

Julia Wrappers for the arpack-ng Fortran library
https://arpack.julialinearalgebra.org/stable
MIT License
69 stars 29 forks source link

Installation error when Julia is built from source #5

Closed yurivish closed 5 years ago

yurivish commented 6 years ago

I tried installing Arpack on 0.7 beta and saw this error trying to run eigs:

julia> A = Diagonal(1:4);
WARNING: Base.Diagonal is deprecated: it has been moved to the standard library package `LinearAlgebra`.
Add `using LinearAlgebra` to your imports.
 in module Main

julia> λ, ϕ = eigs(A, nev = 2);
ERROR: error compiling saupd: could not load library ""
dlopen(.dylib, 1): image not found
Stacktrace:
 [1] aupd_wrapper(::Any, ::getfield(Arpack, Symbol("#matvecA!#24")){LinearAlgebra.Diagonal{Float64,Array{Float64,1}}}, ::getfield(Arpack, Symbol("##18#25")), ::getfield(Arpack, Symbol("##19#26")), ::Int64, ::Bool, ::Bool, ::String, ::Int64, ::Int64, ::String, ::Float64, ::Int64, ::Int64, ::Array{Float64,1}) at /Users/yurivish/.julia/packages/Arpack/Rkbg/src/libarpack.jl:42
 [2] #_eigs#17(::Int64, ::Int64, ::Symbol, ::Float64, ::Int64, ::Nothing, ::Array{Float64,1}, ::Bool, ::Any, ::Any, ::Any) at /Users/yurivish/.julia/packages/Arpack/Rkbg/src/Arpack.jl:176
 [3] (::getfield(Arpack, Symbol("#kw##eigs")))(::NamedTuple{(:nev,),Tuple{Int64}}, ::typeof(eigs), ::LinearAlgebra.Diagonal{Int64,UnitRange{Int64}}) at ./none:0
 [4] top-level scope at none:0

To see if this was solved on master, I tried installing it. But this happens when I try to develop Arpack#master:

Error: Error building `Arpack`:
│ [ Info: Downloading https://github.com/JuliaLinearAlgebra/ArpackBuilder/releases/download/v3.5.0-0.2.20/ArpackBuilder.x86_64-apple-darwin14.tar.gz to /Users/yurivish/.julia/dev/Arpack/deps/usr/downloads/ArpackBuilder.x86_64-apple-darwin14.tar.gz...
│ ┌ Warning: `wait(t::Task)` is deprecated, use `fetch(t)` instead.
│ │   caller = macro expansion at OutputCollector.jl:63 [inlined]
│ └ @ Core OutputCollector.jl:63
│ ┌ Warning: `wait(t::Task)` is deprecated, use `fetch(t)` instead.
│ │   caller = wait(::OutputCollector) at OutputCollector.jl:158
│ └ @ BinaryProvider OutputCollector.jl:158
│ ┌ Warning: `wait(t::Task)` is deprecated, use `fetch(t)` instead.
│ │   caller = wait(::OutputCollector) at OutputCollector.jl:159
│ └ @ BinaryProvider OutputCollector.jl:159
│ ┌ Warning: `wait(t::Task)` is deprecated, use `fetch(t)` instead.
│ │   caller = wait(::OutputCollector) at OutputCollector.jl:163
│ └ @ BinaryProvider OutputCollector.jl:163
│ ERROR: LoadError: LibraryProduct(nothing, ["libarpack"], :libarpack, "Prefix(/Users/yurivish/.julia/dev/Arpack/deps/usr)") is not satisfied, cannot generate deps.jl!
│ Stacktrace:
│  [1] error at ./error.jl:33 [inlined]
│  [2] #write_deps_file#134(::Bool, ::Function, ::String, ::Array{LibraryProduct,1}) at /Users/yurivish/.julia/packages/BinaryProvider/2Hlv/src/Products.jl:389
│  [3] write_deps_file(::String, ::Array{LibraryProduct,1}) at /Users/yurivish/.julia/packages/BinaryProvider/2Hlv/src/Products.jl:376
│  [4] top-level scope at none:0
│  [5] include at ./boot.jl:317 [inlined]
│  [6] include_relative(::Module, ::String) at ./loading.jl:1075
│  [7] include(::Module, ::String) at ./sysimg.jl:29
│  [8] include(::String) at ./client.jl:393
│  [9] top-level scope at none:0
│ in expression starting at /Users/yurivish/.julia/dev/Arpack/deps/build.jl:40
julia> versioninfo()
Julia Version 0.7.0-beta.0
Commit f41b1ecaec (2018-06-24 01:32 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin16.7.0)
  CPU: Intel(R) Core(TM) i7-3615QM CPU @ 2.30GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, ivybridge)
Environment:
  JULIA_EDITOR = subl
aytekinar commented 5 years ago

I think this choice has nothing to do with the so naming of the library. It affects the symbol names, but does not automagically change the library name. Moreover, SUFFIX64_UNDERSCORE does not seem to be an option for building OpenBLAS, either. One needs to parse CMakeLists.txt manually, as cmake-gui ../ does not show it as an option --- neither normal, nor advanced.

andreasnoack commented 5 years ago

See https://github.com/JuliaLang/julia/pull/8734 for some details.

arnonerba commented 5 years ago

Interesting. I tested @chriselrod's workaround and received the same error as @crstnbr:

error compiling aupd_wrapper: error compiling naupd: could not load library "/home/ec2-user/.julia/packages/Arpack/UiiMc/deps/usr/lib/libarpack.so"
libopenblas64_.so.0: cannot open shared object file: No such file or directory

It looks like libarpack.so is indeed trying to link to libopenblas64_.so.0, which only the OpenBLAS version of Julia provides:

$ ldd /home/ec2-user/.julia/packages/Arpack/UiiMc/deps/usr/lib/libarpack.so
    linux-vdso.so.1 (0x00007ffdc23e8000)
    libopenblas64_.so.0 => not found
    libgfortran.so.4 => /lib64/libgfortran.so.4 (0x00007f98334fc000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f98331b1000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f9832f9a000)
    libquadmath.so.0 => /lib64/libquadmath.so.0 (0x00007f9832d5a000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f98329a4000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f9833b30000)

On my system (Amazon Linux 2) there is no libopenblas.so in /usr/lib to symlink to. It looks like Julia is providing that shared library itself when I build it from source with OpenBLAS.

Also, both my OpenBLAS and MKL versions are self-compiled. Arpack builds and tests without error on the OpenBLAS version.

Irratzo commented 5 years ago

The following resolved this 'Arpack Error problem' for my purpose (installing DifferentialEquations.jl which needs Arpack, for use in jupyter) for my setup (precompiled binary package julia 1.0.3; anaconda environment 'julia' with installed julia 1.0.2, conda 4.5.12). It boils down to using ] add IJulia InstantiateFromURL; precompile for the installation.

Now I can finally use DifferentialEquaions in jupyter lab. Note that both julia installations here seem to use the same ~/.julia. Seems that the InstantiateFromURL; precompile command resolves some crucial dependencies more intelligently.

affans commented 5 years ago

Is there a fix for this yet? I tried some suggestions above but they didn't work. I am not sure how the Julia on our university cluster was installed. I've emailed the admin and he'll let me know shortly.

Here is my error message. I don't seem to have libopenblas.so anywhere on my system. If our Julia installation was compiled from source, I am sure it used OpenBLAS (as far as I know, you need to pay for MKL which we didn't do).

(v1.0) pkg> build Arpack
  Building Arpack → `~/.julia/packages/Arpack/qof0w/deps/build.log`
┌ Error: Error building `Arpack`: 
│ ERROR: LoadError: LibraryProduct(nothing, ["libarpack"], :libarpack, "Prefix(/home/affans/.julia/packages/Arpack/qof0w/deps/usr)") is not satisfied, cannot generate deps.jl!
│ Stacktrace:
│  [1] #write_deps_file#152(::Bool, ::Function, ::String, ::Array{LibraryProduct,1}) at /home/affans/.julia/packages/BinaryProvider/4F5Hq/src/Products.jl:414
│  [2] (::getfield(BinaryProvider, Symbol("#kw##write_deps_file")))(::NamedTuple{(:verbose,),Tuple{Bool}}, ::typeof(write_deps_file), ::String, ::Array{LibraryProduct,1}) at ./none:0
│  [3] top-level scope at none:0
│  [4] include(::String) at ./client.jl:392
│  [5] top-level scope at none:0
│ in expression starting at /home/affans/.julia/packages/Arpack/qof0w/deps/build.jl:74
└ @ Pkg.Operations /builddir/build/BUILD/julia/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1097

Some additional information

[affans@hpc ~]$ ldd ./.julia/packages/Arpack/UiiMc/deps/usr/lib/libarpack.so
    linux-vdso.so.1 =>  (0x00002aaaaaacd000)
    libopenblas64_.so.0 => not found
    libgfortran.so.3 => /lib64/libgfortran.so.3 (0x00002aaaaaf26000)
    libm.so.6 => /lib64/libm.so.6 (0x00002aaaab248000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002aaaab54a000)
    libquadmath.so.0 => /lib64/libquadmath.so.0 (0x00002aaaab760000)
    libc.so.6 => /lib64/libc.so.6 (0x00002aaaab99c000)
    /lib64/ld-linux-x86-64.so.2 (0x00002aaaaaaab000)
staticfloat commented 5 years ago

@affans let's figure out some basic information about your Julia. Please post the following:

julia> versioninfo()
Julia Version 1.0.3
Commit 099e826241 (2018-12-18 01:34 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i5-6600 CPU @ 3.30GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, skylake)

julia> Base.libblas_name
"libopenblas64_"

julia> using Libdl; filter(lib -> occursin("blas", lib), Libdl.dllist())
1-element Array{AbstractString,1}:
 "/home/sabae/dist/julia-1.0.3/bin/../lib/julia/libopenblas64_.so"

This will tell us what version of Julia you're using and also what BLAS library you have loaded.

affans commented 5 years ago

Hi @staticfloat Thanks for the help! 

julia> versioninfo()
Julia Version 1.0.3
Commit 04330c0378 (2018-12-16 21:23 UTC)
Platform Info:
  OS: Linux (x86_64-redhat-linux)
  CPU: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, broadwell)
julia> Base.libblas_name
"libopenblasp64_"

Interesting that i have a p64 appended to this.

julia> using Libdl; filter(lib -> occursin("blas", lib), Libdl.dllist())
1-element Array{AbstractString,1}:
 "/usr/bin/../lib64/julia/libopenblasp64_.so"
staticfloat commented 5 years ago

Yes, that would be the problem. Your system's libopenblas is named differently from what Julia by default names it. You can try creating a symlink from libopenblasp64_.sp to libopenblas64_.so, within /usr/lib64/julia, so that either name works. If you don't have write access to /usr/lib64/julia, create a directory in your home (e.g. ~/lib64), place the symlink there (ln -s /usr/lib64/julia/libopenblasp64_.so ~/lib64/libopenblas64_.so), then tack that location onto the end of LD_LIBRARY_PATH and launch Julia (LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$(echo ~)/lib64 julia)

affans commented 5 years ago

@staticfloat That's I'll try that in a second. A quick question though. The file libopenblasp64_.sp already exists in /usr/lib64/julia.. i.e. the julia folder. I am assuming that it is the installation of Julia that makes that folder and puts the libraries there. So how come it can't find it's own file (or that it uses a different name?)

Unless the file libopenblasp64.so comes from somewhere else?

staticfloat commented 5 years ago

The problem is that libarpack.so is looking for libopenblas64_so., not libopenblasp64_.so, which is what Julia has provided. In a "default" Julia installation, the name of libopenblas is standardized to libopenblas64_.so (e.g. no p, which for historical reasons is added on to the end of libopenblas depending on the architecture it was built on). Something about the environment that your Julia was built within does not have this renaming which is not a problem for Julia, but is a problem for ARPACK, because unfortunately binary dependencies are not as flexible as Julia itself, which can handle dynamically changing which library path it looks for on-the-fly. And so we must create a second mapping to allow your libarpack to get "forwarded" to the libopenblas that already exists on your system when it looks for it.

So I suggest that you create a symlink named libopenblas64_.so that points to libopenblasp64_.so, so that when libarpack.so attempts to load libopenblas64_.so, it finds the symlink and follows it to libopenblasp64_.so.

staticfloat commented 5 years ago

Oh, perhaps another more fundamental explanation is helpful; the dynamic linker keys off of the filename of the library that a dynamic object (like libarpack.so) declares as a dependency; meaning that when libarpack.so says it needs libopenblasp64_.so, it completely ignores the existence of libopenblas64_.so because the names are different; from the perspective of the dynamic linker, those two libraries are completely different, even if they would have the same contents.

affans commented 5 years ago

I don't think it's working.. unless I messed up the LD_LIBRARY_PATH. 

(v1.0) pkg> build Arpack
  Building Arpack → `~/.julia/packages/Arpack/qof0w/deps/build.log`
┌ Error: Error building `Arpack`: 
│ ERROR: LoadError: LibraryProduct(nothing, ["libarpack"], :libarpack, "Prefix(/home/affans/.julia/packages/Arpack/qof0w/deps/usr)") is not satisfied, cannot generate deps.jl!
│ Stacktrace:
│  [1] #write_deps_file#152(::Bool, ::Function, ::String, ::Array{LibraryProduct,1}) at /home/affans/.julia/packages/BinaryProvider/4F5Hq/src/Products.jl:414
│  [2] (::getfield(BinaryProvider, Symbol("#kw##write_deps_file")))(::NamedTuple{(:verbose,),Tuple{Bool}}, ::typeof(write_deps_file), ::String, ::Array{LibraryProduct,1}) at ./none:0
│  [3] top-level scope at none:0
│  [4] include(::String) at ./client.jl:392
│  [5] top-level scope at none:0
│ in expression starting at /home/affans/.julia/packages/Arpack/qof0w/deps/build.jl:74
└ @ Pkg.Operations /builddir/build/BUILD/julia/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1097

here is my LD_LIRBARY_PATH:

[affans@hpc lib64]$ echo $LD_LIBRARY_PATH 
/cm/shared/apps/openmpi/gcc/64/1.10.1/lib64:/home/affans/lib64

[affans@hpc lib64]$ ls -l
total 4
lrwxrwxrwx 1 affans abmlab  35 Jan  3 17:13 libopenblas64_.so -> /usr/lib64/julia/libopenblasp64_.so

Maybe I need to "flush" everything and start over. Can I delete my .julia folder with significant problems? I only have a handful of packages installed that I can reinstall. I don't know if ~/.julia contains other important bits of Julia. Thanks for the explanation. I am learning quite a bit!!! Not a linux user :(

staticfloat commented 5 years ago

This issue is getting a little bit long, and since the fundamental technical problem with this package that started this issue has been solved, I'm going to close the issue so that we don't bother too many other users with our problems.

Please copy the above message into a new issue, ping me, and I'll help you work through the problems.

eli-schwartz commented 5 years ago

since the fundamental technical problem with this package that started this issue has been solved

But it isn't solved. It's been partially worked around under certain specific conditions. And the binary mappings you do provide do not include anything suitable for source installations where the default blas implementation is not libopenblas64_.so, but libopenblas.so, or where mkl is in use.

This means that you only support a limited subset of the use cases that julia itself supports, a problem which could have been elegantly solved 6 months ago by making a policy decision that users should be allowed to make one last Hail Mary attempt to use the system arpack library (rather than steadfastly refusing to allow users the right to control their own software).

staticfloat commented 5 years ago

And the binary mappings you do provide do not include anything suitable for source installations where the default blas implementation is not libopenblas64_.so, but libopenblas.so

It is infeasible for us to provide precompiled .so files for all the variations of BLAS library names that could exist in the wild; there are an infinite number. We provide precompiled binaries that will work as long as users are building with the recommended defaults.

a problem which could have been elegantly solved 6 months ago by making a policy decision that users should be allowed to make one last Hail Mary attempt to use the system arpack library (rather than steadfastly refusing to allow users the right to control their own software).

I understand you feel strongly about this; but I'm not sure what more you're asking for. Users can always compile their own libarpack.so and copy it into (for example) ~/.julia/packages/Arpack/<pkg hash>/deps/usr/lib where it will be picked up by a pkg> build Arpack. That works, and serves as the release valve for users that truly need to build their own binaries, so a 'Hail Mary' attempt is always possible. We have attempted to support packaging systems that are much more eager to push users toward building from source on their own machines, and it doesn't work as well as what we have right now, even with the shortcomings that currently exist.

simonbyrne commented 5 years ago

Users can always compile their own libarpack.so and copy it into (for example) ~/.julia/packages/Arpack/<pkg hash>/deps/usr/lib where it will be picked up by a pkg> build Arpack.

Would be good to document this, or throw an informative error message.

staticfloat commented 5 years ago

This is true for all Pkg packages; it's one of those implicit pieces of knowledge that isn't really documented anywhere. Once we get https://github.com/JuliaLang/Pkg.jl/issues/841 sorted out (this is high on my list of priorities) this should become a little more "explicit". We plan to have binary packages be dealt with a little more explicitly than they are right now, with Pkg understanding the concept of a "binary artifact" that is downloaded and reasoned within the Pkg universe just like Julia packages are. Just like you can take manual control of your Julia packages with dev, you will be able to take manual control of binary packages by dev'ing them (or some other, similar verb), instructing Pkg to just load whatever is within ~/.julia/dev/Arpack_jll or somesuch without bothering to engage in any kind of version/compatibility resolution.

Between now and then, any improvements to documentation or error messages are welcome, although I think the right place for this kind of thing is probably within Pkg.jl, not Arpack.jl.

heroxbd commented 5 years ago

@staticfloat I am realy disappointed on how Julia developers take this issue. You are bundling everything, not only arpack, into the user's HOME directory without even attempting to reuse from the host system. That will be a big burden for distribution packagers, say Debian, Gentoo, Arch or Fedora to integrate Julia into a larger operating systems.

It is infeasible for us to provide precompiled .so files for all the variations of BLAS library names that could exist in the wild; there are an infinite number.

Build systems (GNU autotools, CMake, etc.), are designed to deal with the infinite number possibility by allowing users to pass switches to specify which arpack to use.

Repeatedly pointing power users to download big blobs of precompiled binary and bundling everything feels completely unprofessional.

pengwyn commented 5 years ago

I agree with @heroxbd. I think it is great to provide a fallback in the case that the host operating system doesn't support a particular library. However, including popular libraries as binaries is weird. If you absolutely insisted on boxing an external library with your program, wouldn't the normal thing to do here be to compile libarpack? I suspect this would take a long time to compile... but isn't that exactly why we have distro repositories?

What is really annoying is that "building from source" is not a power user thing, but happens in the packaging of Julia for many (most?) distros of Linux. And what's worse is there is no way for the distro maintainer to control what happens when Pkg.add("Arpack") is run.

In short, why not leave inter-dependencies of libraries to the OS and merely complain (or give a link to a binary to download) when a library doesn't exist?

CommunicationAnimale commented 5 years ago

On Fedora 29, after installing the package openblas-devel (with dnf install openblas-devel), I could build Arpack.jl without issues. I found this package by running

$ dnf whatprovides "*/libopenblas64.so"
Last metadata expiration check: 0:16:10 ago on Thu 24 Jan 2019 12:40:05 PM CET.
openblas-devel-0.3.5-1.fc29.x86_64 : Development
     ...: headers and libraries for OpenBLAS
Repo        : updates
Matched from:
Filename    : /usr/lib64/libopenblas64.so

By default, Julia ships with libopenblasp64.so on Fedora.

jamesonquinn commented 5 years ago

I understand why this bug is closed, but I do want to say it's still an issue for ordinary people. I'm trying to run something on a university cluster and just enabling the prepackaged lapack doesn't get this working. I'm working through variations of the suggestions above, and I'll post when I get this fixed, but it's not easy.

In other words — this shouldn't take priority over open bugs, but if there is a chance for a better fix in the future, that would still be great.

....

OK, it seems I've resolved this issue, though my code still isn't running. I ended up dowloading the arpack rpm, extract / configure / make, and symlinking, as suggested by staticfloat above (https://github.com/JuliaLinearAlgebra/Arpack.jl/issues/5#issuecomment-451306576). Now I'm getting a different and probably unrelated error: julia: symbol lookup error: /n/sw/intel-cluster-studio-2017/mkl/lib/intel64/libmkl_intel_thread.so: undefined symbol: omp_get_num_procs

WT215 commented 5 years ago

Is there a fix for this yet? I tried some suggestions above but they didn't work. I am not sure how the Julia on our university cluster was installed. I've emailed the admin and he'll let me know shortly.

Here is my error message. I don't seem to have libopenblas.so anywhere on my system. If our Julia installation was compiled from source, I am sure it used OpenBLAS (as far as I know, you need to pay for MKL which we didn't do).

(v1.0) pkg> build Arpack
  Building Arpack → `~/.julia/packages/Arpack/qof0w/deps/build.log`
┌ Error: Error building `Arpack`: 
│ ERROR: LoadError: LibraryProduct(nothing, ["libarpack"], :libarpack, "Prefix(/home/affans/.julia/packages/Arpack/qof0w/deps/usr)") is not satisfied, cannot generate deps.jl!
│ Stacktrace:
│  [1] #write_deps_file#152(::Bool, ::Function, ::String, ::Array{LibraryProduct,1}) at /home/affans/.julia/packages/BinaryProvider/4F5Hq/src/Products.jl:414
│  [2] (::getfield(BinaryProvider, Symbol("#kw##write_deps_file")))(::NamedTuple{(:verbose,),Tuple{Bool}}, ::typeof(write_deps_file), ::String, ::Array{LibraryProduct,1}) at ./none:0
│  [3] top-level scope at none:0
│  [4] include(::String) at ./client.jl:392
│  [5] top-level scope at none:0
│ in expression starting at /home/affans/.julia/packages/Arpack/qof0w/deps/build.jl:74
└ @ Pkg.Operations /builddir/build/BUILD/julia/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1097

Some additional information

[affans@hpc ~]$ ldd ./.julia/packages/Arpack/UiiMc/deps/usr/lib/libarpack.so
  linux-vdso.so.1 =>  (0x00002aaaaaacd000)
  libopenblas64_.so.0 => not found
  libgfortran.so.3 => /lib64/libgfortran.so.3 (0x00002aaaaaf26000)
  libm.so.6 => /lib64/libm.so.6 (0x00002aaaab248000)
  libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002aaaab54a000)
  libquadmath.so.0 => /lib64/libquadmath.so.0 (0x00002aaaab760000)
  libc.so.6 => /lib64/libc.so.6 (0x00002aaaab99c000)
  /lib64/ld-linux-x86-64.so.2 (0x00002aaaaaaab000)

Same problem here... also tried to deploy julia in university's cluster, but failed to install Pkg.add("Distributions")

pdebuyl commented 5 years ago

Hi all,

I arrived here by googling "julia libopenblas64_.so" or closely related request.

My system:


$ uname -a
Linux my_hostname 4.19.0-1-amd64 #1 SMP Debian 4.19.13-1 (2018-12-30) x86_64 GNU/Linux
$ julia --version
julia version 1.0.3
$ gfortran --version
GNU Fortran (Debian 7.2.0-19) 7.2.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ ldd ~/.julia/packages/Arpack/UiiMc/deps/usr/lib/libarpack.so.2.0.0 
        linux-vdso.so.1 (0x00007ffecf9ee000)
        libopenblas64_.so.0 => /home/username/.julia/packages/Arpack/UiiMc/deps/usr/lib/libopenblas64_.so.0 (0x00007f51112fa000)
        libgfortran.so.5 => /usr/lib/x86_64-linux-gnu/libgfortran.so.5 (0x00007f5111054000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5110ed1000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5110eb7000)
        libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007f5110e76000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5110cb5000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5110c92000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f5113737000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5110a74000)

The problem, as with others, was a missing libopenblas64_.so. I made a symlink in ~/.julia/packages/Arpack/UiiMc/deps/usr/lib/ to the system libopenblas.so (that is actually /usr/lib/x86_64-linux-gnu/libopenblasp-r0.3.5.so on my machine) on it works. I followed the hack here: https://wiki.archlinux.org/index.php/Julia

I do find it hackish and would love to have julia just "pick" the libopenblas.so from my system. libarpack was built properly. I made sure that the system blas matched the current gfortran executable for the version of libgfortran though.

I leave this message here hoping it helps other to get arpack installed.

ayan-b commented 5 years ago

In continuation to the above comment, in order to create the symbolic link one can use the following command:

ln -s /usr/lib/libopenblas.so /usr/lib/libopenblas64_.so.0

(taken from the same archlinux page).

blegat commented 5 years ago

@ayan-b This won't work as soon as arpack call blas functions as the arpack built expect the blas function to be suffixed by 64_. The arpack library shipped by pacman is now v2.1.0 while this package expects 2.0.0 so copying the /usr/lib/libarpack.so.2.1.0 won't work either. Where is what I have done that works, download the v3.5.0 of Arpack and build it as explained in their README (without the make install part). Then do

$ cp .../arpack-ng-3.5.0/SRC/.libs/libarpack.so.2.0.0 ~/.julia/packages/Arpack/UiiMc/deps/usr/lib/
$ julia -e 'import Pkg; Pkg.build("Arpack")'
  Building Arpack → `~/.julia/packages/Arpack/UiiMc/deps/build.log`

Note: In ArpackBuilder, it is written that we need v3.5.0-3 at commit b095052372aa95d4281a645ee1e367c28255c947; see https://github.com/JuliaLinearAlgebra/ArpackBuilder/blob/ed55085cd647b39fefb923c3dd95b03591522760/build_tarballs.jl#L8-L14 but I couldn't find this commit, https://github.com/JuliaLinearAlgebra/ArpackBuilder/commit/b095052372aa95d4281a645ee1e367c28255c947 gives 404. Using the v3.5.0 release as explained above seems to work for the moment.

getzze commented 5 years ago

So far, the only solution to this problem would be to have a distro-specific package for Arpack using ideas from https://github.com/JuliaLang/julia/issues/30528

casparvl commented 5 years ago

@staticfloat I have had a user who also encountered this issue. I am on the support staff of a national compute infrastructure, and because we want users to make the most optimal usage of our hardware, we spend extensive effort to optimize our software installations and in particular our low-level libraries such as OpenBLAS and arpack. Thus, it is important that we can link against those installations already present on the system. Moreover, whenever we can, we build our software from source with appropriate optimization flags for our systems, in order to ensure optimal performance.

Thus, I fully agree with @heroxbd that it is disappointing to see that the Julia developers do not support the use of libraries already present on the system (at least, that is what I understand from the build notes on https://github.com/JuliaLang/julia/blob/master/README.md). To me, it does not seem to align well with the design goal of Julia to be an easy-to-program and at the same time high-performance programming language. Like @heroxbd mentions, there are many (standard) build systems available to deal exactly with the kind of issues that you are mentioning (i.e. various namings of BLAS libraries). Moreover, using a configure step such as those implemented in a standard build systems makes building from source much easier than manually editing Make.inc-files. The use of standard build tools avoids your (power-)users from having to read the manual just to get your software installed.

In conclusion, until the usage of external libraries is better supported by Julia, I have decided to no longer support our system installation. I understand that development is always a matter of choices on how to spend ones time, and I do not mean this post to be 'just a rant'. Rather, I hope that responses such as that from @heroxbd and myself can help evaluate what factors are important to Julia users, especially those on bigger, centrally managed systems such as cluster computers.

StefanKarpinski commented 5 years ago

One concrete issue is that Julia has certain requirements for how BLAS and Arpack are built and those requirements are often not satisfied by system libraries. We could configure Julia to work around the inconsistencies of system libraries on various systems but that just forces those inconsistencies onto our users, which we do not want to do. We have chosen instead of ship our own versions of such libraries so that we provide a maximally consistent and portable experience to our users—and they do appreciate it. Because of this philosophy, Julia code runs very much the same everywhere. The Celeste project, for example was ported from running on Cori to running on Microsoft Azure in less than one developer-week. That would be unthinkable with your traditional HPC application that ends up being deeply coupled to the precise system it was built to run on. A large part of why that was possible was the philosophy that Julia has of trying to provide the same programming experience everywhere.

We're not unwilling to support this kind of use case but doing so is a fair amount of extra work and there is only so much time, money and manpower available so we have to pick and choose our battles. The battle we have chosen to fight first is to make Julia "just work" for the vast majority of users who do not run their code on supercomputers or need to link against specially tuned system libraries. This approach provides the most benefit to the most people. And indeed, Julia users love its package manager because they install packages and things mostly do just work.

However funding is not a fixed thing. If the national labs have special requirements, perhaps there is funding available to help get the work done to support their special use cases. There are companies (Julia Computing is one such) who can help with this kind of work and are more than willing to upstream any improvements made to Julia's build systems to the open source project.

Keno commented 5 years ago

We actively support julia users on several top HPC systems and I personally have talked extensively with the relevant sysadmins about requirements. As a result, I do think we understand the challenges involved in getting Julia running on systems of these sorts and indeed supporting this was an explicit design goal of the Pkg and BinaryBuilder work. However, that feature is not complete yet and will certainly not be started until after we have resolved https://github.com/JuliaLang/Pkg.jl/issues/841. Our focus right now is to make julia work out of the box on as many systems as possible in order to target the most number of users possible, including users on HPC systems. We have made enormous progress in this regard to the point that julia does tend to "just work" in most places. Yes, that means that it is currently harder to support centrally administered software repositories (whether on HPC systems or in Linux distros, etc), but that is simply an indication that we haven't gotten to that part yet, because it's fairly niche for our user community (especially as julia does tend to work for these users, even if using slightly suboptimal library installations which may or may not matter). We will be able to resolve them in due time, but in the mean time, additional suggestions that we don't know what we're doing or don't care about our users are not particularly productive.

heroxbd commented 5 years ago

@StefanKarpinski @Keno Thank you for your constructive responses. I understand your point now. Would you please leave this issue open, so that potential GNU/Linux distribution developers and users could find this bug easily? Denying the existance of this bug puts contributions off.

In the free software community, people volunteer: funding is a good-to-have but not a must to work. I have seen heroic efforts from @cdluminate to introduce Julia to the Debian community, with millions of potential users. I could vision we work toward upstreaming @cdluminate's patches to honestly solve this bug. We care about Julia and we are willing to help.

@Keno, just to comment that it is not "slightly suboptimal", the suboptimal library could easily drag performance down by 10x, e.g. in @casparvl's case.

StefanKarpinski commented 5 years ago

I have never seen a 10x difference in performance because of the way a library was configured or built—frankly, that’s a bit implausible. If that’s the case then either the default we ship should be much better or the “optimized” version is doing something very sketchy and wrong.

heroxbd commented 5 years ago

I have never seen a 10x difference in performance because of the way a library was configured or built—frankly, that’s a bit implausible. If that’s the case then either the default we ship should be much better or the “optimized” version is doing something very sketchy and wrong.

Stefan, there is no need to limit our discussion within reconfiguring and rebuilding a library. This issue cares more about substituting a generic Julia-dependent library with a much more specialized, fine-tuned and optimized one. For example, for netlib blas-reference vs BLIS, the difference will be even more than 10x on exotic platforms like Intel Xeon Phi KNL. In general a 10x difference is not an exaggeration. Scientists and engineers in supercomputing centers work full time to tune performance for a reason.

Apart from performance, portability and easiness for building from source is even more important to users.

StefanKarpinski commented 5 years ago

C'mon, man. Nobody is arguing that Julia users should use reference BLAS. Julia has never—even from the very earliest days—shipped with a reference BLAS because we're not incompetent and we care deeply about performance. A huge amount of work has been done to find and correctly configure high-quality, high-performance, portable libraries like OpenBLAS, dSFMT, etc. We even went so far as creating our own openlibm library because no existing libm at the time met the requirements of being fast, portable, open source and liberally licensed (since then many other projects have adopted openlibm). So talking about stuff like reference BLAS is just a straw man argument and a lecture about "scientists and engineers in supercomputing centers work full time to tune performance for a reason" is just obnoxious and unwarranted. Regarding Xeon Phi KNL, yes, that's what Cori has and @keno got Julia running at 1.5 petaflops on that machine for the Celeste project. So we're well aware of what it takes to get Julia running on HPC systems. It's fine to discuss how to make it easier to use custom libraries on HPC systems but please stick to the actual situation rather than bogus claims that what we are shipping by default is 10x slower.

casparvl commented 5 years ago

@StefanKarpinski @Keno Thanks a lot for your constructive responses. First of all,

additional suggestions that we don't know what we're doing or don't care about our users are not particularly productive.

if that was how my post came accross, I apologize. I know that developers in general care a lot about their projects. Earlier in this thread, there was an argument that 'there are so many names for BLAS libraries', suggesting that this would be the problem - and that, I could not understand. But clearly, that is not the main issue. The main issue as I understand it now seems to be

  1. Technical

Julia has certain requirements for how BLAS and Arpack are built

and although I don't personally grasp the details of this statement (don't these libs have standard APIs?) I trust you know far better than I do.

  1. Design choice

We have chosen instead of ship our own versions of such libraries so that we provide a maximally consistent and portable experience to our users—and they do appreciate it.

This, clearly, is a fair design choice and far be it from me to argue with that. Apart from performance, I also encounter users with portability issues, so I understand the value of this design choice.

I know that a lot of these things are a matter of priority, because resources are always limited. And, one key factor in determining priorities is to know what is important to your users (the exact same applies to us, as an HPC center by the way). I would like my above post to be a signal that yes, linking to low level libraries could be important to a (group of) users, namely those on university clusters & HPC systems. They may not be your biggest or most important users group - you probably know that better than I do - but running on such systems has some weird implications that do not apply to 'normal' users.

For example, I do not assume that linking to system libraries will result in a 10x speedup in a typical case, but what you have to realize is that on a typical HPC system, if one can achieve a 5% speedup of all the software running on that system, that means one could save anywhere between tens of thousands and millions of USD/Euro in hardware, depending on the size of the system. Thus, even though a lot of software download instructions say 'just download the binaries, because it is easier', the HPC community tends to be willing to spend the extra effort and compile from source. In 99/100 cases, this is worthwhile because software developers themselves regularly focus on functionality, rather than performance. I'm glad to see that Julia is in the 1%.

wsshin commented 5 years ago

I have a question related to this thread. On one Mac, I have been building Arpack.jl successfully by installing gcc@7 by homebrew and setting DYLD_LIBRARY_PATH as instructed in https://github.com/JuliaLinearAlgebra/Arpack.jl/issues/5#issuecomment-400225495, but on another Mac (with the same macOS version), the same trick does not work.

By inspecting ~/.julia/dev/Arpack/deps/usr/logs/ArpackBuilder.log on both Macs, I noticed the line 

-- The Fortran compiler identification is GNU 7.3.0`

in the first Mac as desired, but 

-- The Fortran compiler identification is GNU 8.1.0`

in the second Mac. Apparently pointing DYLD_LIBRARY_PATH to gcc@7 is not forcing the second Mac to use GCC 7.

Am I missing something? Any ideas?

staticfloat commented 5 years ago

Apparently pointing DYLD_LIBRARY_PATH to gcc@7 is not forcing the second Mac to use GCC 7.

DYLD_LIBRARY_PATH changes which libraries get loaded at runtime, but when invoking a compiler, you need to set PATH to point to where that compiler lives. Try running which -a gfortran to see where all the gfortran's are that live on your machine, and set PATH to point to your gfortran-7. (You may also be able to set export FC=/path/to/gfortran-7).

wsshin commented 5 years ago

@staticfloat, on both Macs the directory of the homebrew-installed gfortran is the top PATH directory. I also tried export FC=/path/to/gfortran-7 but it didn't work.

I even tried to link gfortran to gfortran-7, but the build log on the problematic Mac still says

-- The Fortran compiler identification is GNU 8.1.0

and I did pkg> build Arpack after pkg> rm Arpack, removing the ~/.julia/dev/Arpack directory, and pkg> dev Arpack to make sure that any build configurations from the previous build attempt does not affect the current build attempt.

Any ideas how to force the Fortran version to use for building Arpack?

wsshin commented 5 years ago

Any ideas?

staticfloat commented 5 years ago

@wsshin So that we don't bother other people, let's open a new issue. I will briefly note that I do not think you are actually compiling anything, since the log file location leads me to believe that you are instead getting some precompiled binaries. I'm willing to bet that you compiled julia itself with GCC 8, and so it is downloading GCC 8-compatible binaries for use with Julia. In any case, please open a new issue and give as much information as possible, including how Julia was installed, what system you're running on, what compilers you have installed, etc...

puckvg commented 5 years ago

I understand why this bug is closed, but I do want to say it's still an issue for ordinary people. I'm trying to run something on a university cluster and just enabling the prepackaged lapack doesn't get this working. I'm working through variations of the suggestions above, and I'll post when I get this fixed, but it's not easy.

In other words — this shouldn't take priority over open bugs, but if there is a chance for a better fix in the future, that would still be great.

....

OK, it seems I've resolved this issue, though my code still isn't running. I ended up dowloading the arpack rpm, extract / configure / make, and symlinking, as suggested by staticfloat above (#5 (comment)). Now I'm getting a different and probably unrelated error: julia: symbol lookup error: /n/sw/intel-cluster-studio-2017/mkl/lib/intel64/libmkl_intel_thread.so: undefined symbol: omp_get_num_procs

I'm getting the same error 'julia: symbol lookup error: /home/puck/.julia/packages/Arpack/cu5By/deps/usr/lib/libarpack.so: undefined symbol: dlamch64'

Did you ever fix it?

nalimilan commented 5 years ago

@puckvg What's your configuration?

mauro3 commented 4 years ago

This seems to also happen on Cirrus CI: https://cirrus-ci.com/task/5116177080123392?command=test#L69

wi11dey commented 2 years ago

I've found a workaround for those like @carstenbauer (and myself) who are on university clusters and can't simply make a symlink /usr/lib/libopenblas64_.so to the real location of OpenBLAS:

  1. Go to your ~/.julia folder
  2. find -iname 'libarpack.so to find where the libarpack.so that is trying to reference libopenblas64_.so lives
  3. patchelf --replace-needed libopenblas64_.so libopenblas.so libarpack.so (patchelf is available here if your distro doesn't have it

The process could be upstreamed into Arpack.jl and make more reliable by automatically making the libarpack.so binary reference whatever OpenBLAS_jll.libopenblas is set to, which is correct for every distro's Julia

carstenbauer commented 2 years ago

@wi11dey FWIW, let me say that (at least for production runs) I don't build Julia from source anymore because there is almost no good reason anymore to do so. The main reason for me was to use Intel MKL but that can nowadays be easily achieved with MKL.jl.