search

conda-forge / julia-feedstock

A conda-smithy repository for julia.
BSD 3-Clause "New" or "Revised" License
23 stars 32 forks source link

CHOLMOD Warning, Linking Issues on Mac #274

Open mharradon opened 5 months ago

mharradon commented 5 months ago

Solution to issue cannot be found in the documentation.

Issue

Installing Julia just from conda-forge onto an Intel Mac results in CHOLMOD warning. I am not sure if what it states is true, as apparently this warning has been erroneous in the past:

» conda create --override-channels --channel conda-forge -n julia-test julia
…
» conda activate julia-test
» julia --project=.
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.10.2 (2024-03-01)
 _/ |\__'_|_|_|\__'_|  |  https://github.com/conda-forge/julia-feedstock
|__/                   |

julia> using SparseArrays
┌ Warning: CHOLMOD version incompatibility
│
│ Julia was compiled with CHOLMOD version 4.0.4. It is
│ currently linked with version 3.0.14.
│ This might cause Julia to terminate when working with
│ sparse matrix factorizations, e.g. solving systems of
│ equations with \.
│
│ It is recommended that you use Julia with the same major
│ version of CHOLMOD as the one used during the build, or
│ download the generic binaries from www.julialang.org,
│ which ship with the correct versions of all dependencies.
└ @ SparseArrays.CHOLMOD ~/mambaforge/envs/julia-test/share/julia/stdlib/v1.10/SparseArrays/src/solvers/cholmod.jl:206

When not forcing conda-forge (which I expect to have issues anyway) I get:

julia> using SparseArrays
ERROR: InitError: could not load library "@rpath/libspqr.2.dylib"
dlopen(@rpath/libspqr.2.dylib, 0x0001): Library not loaded: @rpath/libtbb.dylib
  Referenced from: <D2B2A060-F4A6-34EA-BC95-649F75FE4EA1> /Users/mharradon/mambaforge/envs/julia-test/lib/libspqr.2.0.9.dylib
  Reason: tried: '/Users/mharradon/mambaforge/envs/julia-test/lib/libtbb.dylib' (no such file), '/Users/mharradon/mambaforge/envs/julia-test/lib/libtbb.dylib' (no such file), '/Users/mharradon/mambaforge/envs/julia-test/lib/julia/libtbb.dylib' (no such file), '/Users/mharradon/mambaforge/envs/julia-test/lib/libtbb.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/mharradon/mambaforge/envs/julia-test/lib/libtbb.dylib' (no such file), '/Users/mharradon/mambaforge/envs/julia-test/lib/julia/../libtbb.dylib' (no such file), '/Users/mharradon/mambaforge/envs/julia-test/lib/libtbb.dylib' (no such file), '/usr/local/lib/libtbb.dylib' (no such file), '/usr/lib/libtbb.dylib' (no such file, not in dyld cache)
Stacktrace:
  [1] dlopen(s::String, flags::UInt32; throw_error::Bool)
    @ Base.Libc.Libdl ./libdl.jl:117
  [2] dlopen
    @ ./libdl.jl:116 [inlined]
  [3] dlopen(s::String)
    @ Base.Libc.Libdl ./libdl.jl:116
  [4] __init__()
    @ SuiteSparse_jll ~/mambaforge/envs/julia-test/share/julia/stdlib/v1.10/SuiteSparse_jll/src/SuiteSparse_jll.jl:104
  [5] run_module_init(mod::Module, i::Int64)
    @ Base ./loading.jl:1134
  [6] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1122
  [7] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any})
    @ Base ./loading.jl:1067
  [8] _tryrequire_from_serialized(modkey::Base.PkgId, path::String, ocachepath::String, sourcepath::String, depmods::Vector{Any})
    @ Base ./loading.jl:1481
  [9] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128)
    @ Base ./loading.jl:1574
 [10] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1938
 [11] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1812
 [12] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [13] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [14] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1803
 [15] macro expansion
    @ ./loading.jl:1790 [inlined]
 [16] macro expansion
    @ ./lock.jl:267 [inlined]
 [17] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1753
 [18] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [19] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [20] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1746
during initialization of module SuiteSparse_jll

Installed packages

arpack             conda-forge/osx-64::arpack-3.8.0-nompi_hb44a6d1_101
  bzip2              conda-forge/osx-64::bzip2-1.0.8-h10d778d_5
  c-ares             conda-forge/osx-64::c-ares-1.27.0-h10d778d_0
  ca-certificates    conda-forge/osx-64::ca-certificates-2024.2.2-h8857fd0_0
  curl               conda-forge/osx-64::curl-8.7.1-h726d00d_0
  gettext            conda-forge/osx-64::gettext-0.21.1-h8a4c099_0
  git                conda-forge/osx-64::git-2.44.0-pl5321h8687b19_0
  gmp                conda-forge/osx-64::gmp-6.3.0-h73e2aa4_1
  icu                conda-forge/osx-64::icu-73.2-hf5e326d_0
  julia              conda-forge/osx-64::julia-1.10.2-h7cbd117_1
  krb5               conda-forge/osx-64::krb5-1.21.2-hb884880_0
  libblas            conda-forge/osx-64::libblas-3.9.0-21_osx64_openblas
  libcblas           conda-forge/osx-64::libcblas-3.9.0-21_osx64_openblas
  libcurl            conda-forge/osx-64::libcurl-8.7.1-h726d00d_0
  libcxx             conda-forge/osx-64::libcxx-16.0.6-hd57cbcb_0
  libedit            conda-forge/osx-64::libedit-3.1.20191231-h0678c8f_2
  libev              conda-forge/osx-64::libev-4.33-h10d778d_2
  libexpat           conda-forge/osx-64::libexpat-2.6.2-h73e2aa4_0
  libgfortran        conda-forge/osx-64::libgfortran-5.0.0-13_2_0_h97931a8_3
  libgfortran5       conda-forge/osx-64::libgfortran5-13.2.0-h2873a65_3
  libhwloc           conda-forge/osx-64::libhwloc-2.9.3-default_h24e0189_1009
  libiconv           conda-forge/osx-64::libiconv-1.17-hd75f5a5_2
  liblapack          conda-forge/osx-64::liblapack-3.9.0-21_osx64_openblas
  libnghttp2         conda-forge/osx-64::libnghttp2-1.58.0-h64cf6d3_1
  libopenblas        conda-forge/osx-64::libopenblas-0.3.26-openmp_hfef2a42_0
  libopenblas-ilp64  conda-forge/osx-64::libopenblas-ilp64-0.3.26-openmp_h17068ef_0
  libopenlibm4       conda-forge/osx-64::libopenlibm4-0.8.1-h10d778d_1
  libosxunwind       conda-forge/osx-64::libosxunwind-0.0.6-h940c156_0
  libssh2            conda-forge/osx-64::libssh2-1.11.0-hd019ec5_0
  libutf8proc        conda-forge/osx-64::libutf8proc-2.8.0-hb7f2c08_0
  libxml2            conda-forge/osx-64::libxml2-2.12.6-hc0ae0f7_1
  libzlib            conda-forge/osx-64::libzlib-1.2.13-h8a1eda9_5
  llvm-openmp        conda-forge/osx-64::llvm-openmp-18.1.2-hb6ac08f_0
  metis              conda-forge/osx-64::metis-5.1.0-he965462_1007
  mpfr               conda-forge/osx-64::mpfr-4.2.1-h0c69b56_0
  ncurses            conda-forge/osx-64::ncurses-6.4.20240210-h73e2aa4_0
  openblas-ilp64     conda-forge/osx-64::openblas-ilp64-0.3.26-openmp_ha601604_0
  openlibm           conda-forge/osx-64::openlibm-0.8.1-h10d778d_1
  openssl            conda-forge/osx-64::openssl-3.2.1-hd75f5a5_1
  p7zip              conda-forge/osx-64::p7zip-16.02-he49afe7_1001
  pcre2              conda-forge/osx-64::pcre2-10.43-h0ad2156_0
  perl               conda-forge/osx-64::perl-5.32.1-7_h10d778d_perl5
  suitesparse        conda-forge/osx-64::suitesparse-5.10.1-h4bf45ed_3
  tbb                conda-forge/osx-64::tbb-2021.11.0-h7728843_1
  xz                 conda-forge/osx-64::xz-5.2.6-h775f41a_0
  zlib               conda-forge/osx-64::zlib-1.2.13-h8a1eda9_5
  zstd               conda-forge/osx-64::zstd-1.5.5-h829000d_0

Not forcing channel: 

arpack             conda-forge/osx-64::arpack-3.8.0-nompi_hb44a6d1_101
  blas               main/osx-64::blas-1.0-openblas
  bzip2              main/osx-64::bzip2-1.0.8-h6c40b1e_5
  c-ares             conda-forge/osx-64::c-ares-1.27.0-h10d778d_0
  ca-certificates    main/osx-64::ca-certificates-2024.3.11-hecd8cb5_0
  curl               main/osx-64::curl-8.5.0-h04015c4_0
  gdbm               main/osx-64::gdbm-1.18-hdccc71a_4
  gettext            main/osx-64::gettext-0.21.0-he85b6c0_1
  git                conda-forge/osx-64::git-2.44.0-pl5321h8687b19_0
  gmp                conda-forge/osx-64::gmp-6.3.0-h73e2aa4_1
  icu                main/osx-64::icu-73.1-hcec6c5f_0
  julia              conda-forge/osx-64::julia-1.10.2-h7cbd117_1
  krb5               main/osx-64::krb5-1.20.1-h428f121_1
  libblas            conda-forge/osx-64::libblas-3.9.0-21_osx64_openblas
  libcurl            main/osx-64::libcurl-8.5.0-hf20ceda_0
  libcxx             conda-forge/osx-64::libcxx-16.0.6-hd57cbcb_0
  libedit            main/osx-64::libedit-3.1.20230828-h6c40b1e_0
  libev              main/osx-64::libev-4.33-h9ed2024_1
  libexpat           conda-forge/osx-64::libexpat-2.6.2-h73e2aa4_0
  libgfortran        conda-forge/osx-64::libgfortran-5.0.0-13_2_0_h97931a8_3
  libgfortran5       conda-forge/osx-64::libgfortran5-13.2.0-h2873a65_3
  libiconv           conda-forge/osx-64::libiconv-1.17-hd75f5a5_2
  liblapack          conda-forge/osx-64::liblapack-3.9.0-21_osx64_openblas
  libnghttp2         conda-forge/osx-64::libnghttp2-1.58.0-h64cf6d3_1
  libopenblas        conda-forge/osx-64::libopenblas-0.3.26-openmp_hfef2a42_0
  libopenblas-ilp64  conda-forge/osx-64::libopenblas-ilp64-0.3.26-openmp_h17068ef_0
  libopenlibm4       conda-forge/osx-64::libopenlibm4-0.8.1-h10d778d_1
  libosxunwind       conda-forge/osx-64::libosxunwind-0.0.6-h940c156_0
  libssh2            conda-forge/osx-64::libssh2-1.11.0-hd019ec5_0
  libutf8proc        conda-forge/osx-64::libutf8proc-2.8.0-hb7f2c08_0
  libxml2            main/osx-64::libxml2-2.10.4-h1bd7e62_1
  libzlib            conda-forge/osx-64::libzlib-1.2.13-h8a1eda9_5
  llvm-openmp        conda-forge/osx-64::llvm-openmp-18.1.2-hb6ac08f_0
  metis              main/osx-64::metis-5.1.0-h0a44026_4
  mpfr               conda-forge/osx-64::mpfr-4.2.1-h0c69b56_0
  ncurses            main/osx-64::ncurses-6.4-hcec6c5f_0
  openblas-ilp64     conda-forge/osx-64::openblas-ilp64-0.3.26-openmp_ha601604_0
  openlibm           conda-forge/osx-64::openlibm-0.8.1-h10d778d_1
  openssl            conda-forge/osx-64::openssl-3.2.1-hd75f5a5_1
  p7zip              main/osx-64::p7zip-16.02-hcec6c5f_0
  pcre2              conda-forge/osx-64::pcre2-10.43-h0ad2156_0
  perl               main/osx-64::perl-5.34.0-h435f0c2_2
  readline           main/osx-64::readline-8.2-hca72f7f_0
  suitesparse        main/osx-64::suitesparse-5.10.1-h577c77e_2
  tbb                main/osx-64::tbb-2021.8.0-ha357a0b_0
  xz                 main/osx-64::xz-5.4.6-h6c40b1e_0
  zlib               conda-forge/osx-64::zlib-1.2.13-h8a1eda9_5

Environment info

active environment : julia-test
    active env location : /Users/mharradon/mambaforge/envs/julia-test
            shell level : 3
       user config file : /Users/mharradon/.condarc
 populated config files : /Users/mharradon/mambaforge/.condarc
                          /Users/mharradon/.condarc
          conda version : 23.11.0
    conda-build version : 3.28.2
         python version : 3.10.10.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=skylake
                          __conda=23.11.0=0
                          __osx=14.4.1=0
                          __unix=0=0
       base environment : /Users/mharradon/mambaforge  (writable)
      conda av data dir : /Users/mharradon/mambaforge/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/main/osx-64
                          https://conda.anaconda.org/main/noarch
                          https://conda.anaconda.org/conda-forge/osx-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /Users/mharradon/mambaforge/pkgs
                          /Users/mharradon/.conda/pkgs
       envs directories : /Users/mharradon/mambaforge/envs
                          /Users/mharradon/.conda/envs
               platform : osx-64
             user-agent : conda/23.11.0 requests/2.28.2 CPython/3.10.10 Darwin/23.4.0 OSX/14.4.1 solver/libmamba conda-libmamba-solver/23.12.0 libmambapy/1.5.6
                UID:GID : 1647354838:187400128
             netrc file : None
           offline mode : False

Thanks for taking a look!

mkitti commented 5 months ago

A few thoughts:

  1. I only concern myself with the conda-forge channel. If you want Julia support for main, please contact Anaconda, Inc.
  2. This is a dependency issue. I suggest contacting https://github.com/conda-forge/suitesparse-feedstock about packaging version 7.
  3. For a short term fix, I recommend downloading Julia via the instructions https://julialang.org/downloads/ and managing Julia via juliaup.
  4. Another approach would be using the pyjuliacall package, the juliaup package, or the jill package.
mharradon commented 5 months ago

Thanks for the quick reply! To be clear regarding (1), my primary issue here is with running the conda-forge distribution of Julia. I don't install Julia from main in either example (the latter error may be irrelevant due to channel mixing). My primary concern is the warning of improper library linking when running this conda-forge Julia distribution in the most basic installation.

mkitti commented 5 months ago

Again, the problem is that the suitesparse-feedstock is out of date. There's nothing to be done to fix this in this repository.

minrk commented 3 months ago

276 updates to suitesparse 7

mkitti commented 3 months ago

Thanks. I do need to get around to changing the build to adapt for this. Keeping pinging me if this is not fixed.

ianna commented 2 months ago

Hi @minrk It would be great if it's fixed. Thanks!

minrk commented 2 months ago

276 is merged, so if someone can check in an hour or two, we can close this if it's fixed.

orenbenkiki commented 2 months ago

I just encountered https://github.com/JuliaSparse/SparseArrays.jl/issues/542 - is this the same issue?

minrk commented 2 months ago

Can you show conda env export? The latest build is not done with cholmod 4, so I think you aren't getting the build where this is fixed

orenbenkiki commented 2 months ago

Here (clean new environment and then conda install julia):

$ conda env export
name: jupyter
channels:
  - conda-forge
  - intel
  - defaults
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=2_gnu
  - arpack=3.9.1=nompi_h77f6705_101
  - bzip2=1.0.8=hd590300_5
  - c-ares=1.28.1=hd590300_0
  - ca-certificates=2024.6.2=hbcca054_0
  - curl=8.8.0=he654da7_0
  - git=2.45.2=pl5321ha099dd3_1
  - gmp=6.3.0=h59595ed_1
  - icu=73.2=h59595ed_0
  - julia=1.10.4=hf18f99d_1
  - keyutils=1.6.1=h166bdaf_0
  - krb5=1.21.2=h659d440_0
  - libblas=3.9.0=22_linux64_openblas
  - libcblas=3.9.0=22_linux64_openblas
  - libcurl=8.8.0=hca28451_0
  - libedit=3.1.20191231=he28a2e2_2
  - libev=4.33=hd590300_2
  - libexpat=2.6.2=h59595ed_0
  - libgcc-ng=13.2.0=h77fa898_10
  - libgfortran-ng=13.2.0=h69a702a_10
  - libgfortran5=13.2.0=h3d2ce59_10
  - libgit2=1.8.1=he8d1d4c_1
  - libgomp=13.2.0=h77fa898_10
  - libhwloc=2.10.0=default_h5622ce7_1001
  - libiconv=1.17=hd590300_2
  - liblapack=3.9.0=22_linux64_openblas
  - libnghttp2=1.58.0=h47da74e_1
  - libopenblas=0.3.27=pthreads_h413a1c8_0
  - libopenblas-ilp64=0.3.27=pthreads_h384dd9e_0
  - libopenlibm4=0.8.1=hd590300_1
  - libssh2=1.11.0=h0841786_0
  - libstdcxx-ng=13.2.0=hc0a3c3a_10
  - libunwind=1.6.2=h9c3ff4c_0
  - libutf8proc=2.8.0=h166bdaf_0
  - libxcrypt=4.4.36=hd590300_1
  - libxml2=2.12.7=hc051c1a_1
  - libzlib=1.3.1=h4ab18f5_1
  - mbedtls=3.5.1=h59595ed_0
  - metis=5.1.0=h59595ed_1007
  - mpfr=4.2.1=h9458935_1
  - ncurses=6.5=h59595ed_0
  - openblas-ilp64=0.3.27=pthreads_h3d04fff_0
  - openlibm=0.8.1=hd590300_1
  - openssl=3.3.1=h4ab18f5_0
  - p7zip=16.02=h9c3ff4c_1001
  - pcre2=10.44=h0f59acf_0
  - perl=5.32.1=7_hd590300_perl5
  - suitesparse=7.7.0=hf4753ba_1
  - tbb=2021.12.0=h297d8ca_1
  - xz=5.2.6=h166bdaf_0
  - zlib=1.3.1=h4ab18f5_1
  - zstd=1.5.6=ha6fb4c9_0
prefix: /home/obk/anaconda3/envs/jupyter
minrk commented 2 months ago

Yes, I think I am starting to understand. That message doesn't quite mean what it says - SparseArrays expects a particular version of suitesparse that it would usually be compiled against, but doesn't check what it was actually compiled against. It mistakenly claims that this is the version julia is compiled against, which is not true, but it does appropriately warn that the version is significantly different from what it expects. Since it's different by a major version, it's reasonable to warn that things might not work.

Our patch is enough to get the current suitesparse to be loaded and everything seems to work, but this hardcoded version check is still there.