InitError: LightGBM.LibraryNotFoundError and ArgumentError: NULL library handle

ChristianMichelsen commented 2 years ago

Hi!

I am having trouble (installing and) using LightGBM.jl.

I have tried a combination of using just LightGBM.jl, or with an external binary (either homebrew or by building from Github). No matter what, if I open Julia 1.7.3 in a new, fresh environment with only LightGBM.jl added, I get the following:

/work/test ❯ julia
  Activating project at `~/work/test`
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.7.3 (2022-05-06)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using LightGBM
[ Info: lib_lightgbm not found in system dirs, trying fallback
ERROR: InitError: LightGBM.LibraryNotFoundError("lib_lightgbm not found. Please ensure this library is either in system dirs or the dedicated paths: [\"/Users/michelsen/.julia/packages/LightGBM/A7zVd/src\"]")
Stacktrace:
 [1] find_library(library_name::String, custom_paths::Vector{String})
   @ LightGBM ~/.julia/packages/LightGBM/A7zVd/src/LightGBM.jl:32
 [2] __init__()
   @ LightGBM ~/.julia/packages/LightGBM/A7zVd/src/LightGBM.jl:41
 [3] _include_from_serialized(path::String, depmods::Vector{Any})
   @ Base ./loading.jl:768
 [4] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String)
   @ Base ./loading.jl:854
 [5] _require(pkg::Base.PkgId)
   @ Base ./loading.jl:1097
 [6] require(uuidkey::Base.PkgId)
   @ Base ./loading.jl:1013
 [7] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:997
during initialization of module LightGBM

julia> using LightGBM

julia>

That is, when I try to load LightGBM, I first get an error, and then if I try to do load it again, the error disappears. However, when I try to continue with the rest of the example from the readme, I eventually run into the following issue:

julia> fit!(estimator, X_train, y_train, (X_test, y_test))
ERROR: ArgumentError: NULL library handle
Stacktrace:
 [1] #dlsym#1
   @ ./libdl.jl:57 [inlined]
 [2] dlsym
   @ ./libdl.jl:57 [inlined]
 [3] LGBM_DatasetCreateFromMat(data::Matrix{Float64}, parameters::String, is_row_major::Bool)
   @ LightGBM ~/.julia/packages/LightGBM/A7zVd/src/wrapper.jl:101
 [4] fit!(estimator::LGBMClassification, X::Matrix{Float64}, y::Vector{Float64}, test::Tuple{Matrix{Float64}, Vector{Float64}}; verbosity::Int64, is_row_major::Bool, weights::Vector{Float32}, init_score::Vector{Float64}, truncate_booster::Bool)
   @ LightGBM ~/.julia/packages/LightGBM/A7zVd/src/fit.jl:43
 [5] fit!(estimator::LGBMClassification, X::Matrix{Float64}, y::Vector{Float64}, test::Tuple{Matrix{Float64}, Vector{Float64}})
   @ LightGBM ~/.julia/packages/LightGBM/A7zVd/src/fit.jl:39
 [6] top-level scope
   @ REPL[19]:1

julia>

It seems to me that LightGBM.jl does not really "connect" to the "real" LightGBM software.

I have tried adding LightGBM via the Libdl approach and by setting DYLD_LIBRARY_PATH, as pr. the readme's suggestion, but without any luck.

Can someone help me in the right direction?

Right now, lightgbm points to /opt/homebrew/bin/lightgbm, but I have also the LightGBM source at /Users/michelsen/software/LightGBM in case that's relevant.

yaxxie commented 2 years ago

Hi @ChristianMichelsen, thanks for the issue!

So the problem from what I can tell is that the lib fails to load, and this is clear from the original exception. Attempting to load again doesn't re-throw, because it uses the module level init protocol (which is intended for once-runtime loads). Since you effectively ignored the original exception it is not too surprising it doesn't work later on.

This can probably be fixed (the bit where a second import does not re-throw).

As to why the library is not loading: Please can you confirm that you have libomp installed and linkable on the system?

Additionally, you should clarify what you mean by "lightgbm points to /opt/homebrew/bin/lightgbm"? Do you mean to say that the library is in this folder? If so, please confirm that it is there with the filename lib_lightgbm.dylib. Please also confirm that ldd (or whatever the mac equivalent would be) on the file has resolves all link targets.

If the file is present but with a name other than lib_lightgbm.dylib, try creating a symlink in the same folder to file file with that name (and let me know if that was the case, so that we can update the library pointer search for commonly distributed library names)

In general, if you take the vanilla package without any lightgbm library and load it and it throws an exception, it means limomp isn't installed on your system. Please take steps to confirm that it is in fact installed and linkable and report back.

ChristianMichelsen commented 2 years ago

Please can you confirm that you have libomp installed and linkable on the system?

I have run brew install libomp previously, which I assume is enough? Or how do I check that libomp is installed and linkable?

Additionally, you should clarify what you mean by "lightgbm points to /opt/homebrew/bin/lightgbm"?

I mean that when I type lightgbm points to /opt/homebrew/bin/lightgbm. This folder does not contain any lib_lightgbm.dylib.

However, my non-brew installation (at /Users/michelsen/software/LightGBM) contains both the binary lightgbm and lib_lightgbm.so. Yet, when I set DYLD_LIBRARY_PATH to /Users/michelsen/software/LightGBM via Libdl, it does not change anything (compared to the brew-installed path).

yaxxie commented 2 years ago

However, my non-brew installation (at /Users/michelsen/software/LightGBM) contains both the binary lightgbm and lib_lightgbm.so

Compiled from source? I'm surprised you end up with a library with a .so extension rather than a .dylib extension

Could you try to do the following?

ln -s /Users/michelsen/software/LightGBM/lib_lightgbm.so /Users/michelsen/software/LightGBM/lib_lightgbm.dylib
DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH:/Users/michelsen/software/LightGBM/ julia -e "using LightGBM"

and show me what happens?

ChristianMichelsen commented 2 years ago

To be honest, I just blindly followed the installation guide for Mac.

I tried running your command, but I am still getting:

~/wo/r/julia main ⇡2 !5 ?5 ❯ DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH:/Users/michelsen/software/LightGBM/ julia -e "using LightGBM"
  Activating project at `~/work/reference_free/julia`
[ Info: lib_lightgbm not found in system dirs, trying fallback
ERROR: InitError: LightGBM.LibraryNotFoundError("lib_lightgbm not found. Please ensure this library is either in system dirs or the dedicated paths: [\"/Users/michelsen/.julia/packages/LightGBM/A7zVd/src\"]")
Stacktrace:
 [1] find_library(library_name::String, custom_paths::Vector{String})
   @ LightGBM ~/.julia/packages/LightGBM/A7zVd/src/LightGBM.jl:32
 [2] __init__()
   @ LightGBM ~/.julia/packages/LightGBM/A7zVd/src/LightGBM.jl:41
 [3] _include_from_serialized(path::String, depmods::Vector{Any})
   @ Base ./loading.jl:768
 [4] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String)
   @ Base ./loading.jl:854
 [5] _require(pkg::Base.PkgId)
   @ Base ./loading.jl:1097
 [6] require(uuidkey::Base.PkgId)
   @ Base ./loading.jl:1013
 [7] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:997
during initialization of module LightGBM

yaxxie commented 2 years ago

And the output for this?

ls -l /Users/michelsen/.julia/packages/LightGBM/A7zVd/src

ChristianMichelsen commented 2 years ago

~/software/LightGBM master ❯ ls -l /Users/michelsen/.julia/packages/LightGBM/A7zVd/src
.r--r--r-- 4.1k michelsen  8 Jun 15:42  cv.jl
.r--r--r--  13k michelsen  8 Jun 15:42  estimators.jl
.r--r--r--  12k michelsen  8 Jun 15:42  fit.jl
.rw-r--r-- 3.8M michelsen  8 Jun 15:42  lib_lightgbm.dylib
.rwxr-xr-x 3.3M michelsen  8 Jun 15:56  lib_lightgbm.so
.rwxr-xr-x 3.2M michelsen  8 Jun 15:56  lightgbm
.r--r--r-- 1.3k michelsen  8 Jun 15:42  LightGBM.jl
.r--r--r--  16k michelsen  8 Jun 15:42  MLJInterface.jl
.r--r--r-- 2.7k michelsen  8 Jun 15:42  predict.jl
.r--r--r-- 3.3k michelsen  8 Jun 15:42  search_cv.jl
.r--r--r-- 4.0k michelsen  8 Jun 15:42  utils.jl
.r--r--r--  21k michelsen  8 Jun 15:42  wrapper.jl

yaxxie commented 2 years ago

Can you run the following:

otool -l /Users/michelsen/.julia/packages/LightGBM/A7zVd/src/lib_lightgbm.so
otool -l /Users/michelsen/.julia/packages/LightGBM/A7zVd/src/lib_lightgbm.dylib

ChristianMichelsen commented 2 years ago

~/software/LightGBM master ❯ otool -l /Users/michelsen/.julia/packages/LightGBM/A7zVd/src/lib_lightgbm.so
/Users/michelsen/.julia/packages/LightGBM/A7zVd/src/lib_lightgbm.so:
Load command 0
      cmd LC_SEGMENT_64
  cmdsize 712
  segname __TEXT
   vmaddr 0x0000000000000000
   vmsize 0x0000000000204000
  fileoff 0
 filesize 2113536
  maxprot 0x00000005
 initprot 0x00000005
   nsects 8
    flags 0x0
Section
  sectname __text
   segname __TEXT
      addr 0x0000000000002fd8
      size 0x00000000001c8d70
    offset 12248
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x80000400
 reserved1 0
 reserved2 0
Section
  sectname __stubs
   segname __TEXT
      addr 0x00000000001cbd48
      size 0x0000000000000ad4
    offset 1883464
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x80000408
 reserved1 0 (index into indirect symbol table)
 reserved2 12 (size of stubs)
Section
  sectname __init_offsets
   segname __TEXT
      addr 0x00000000001cc81c
      size 0x0000000000000004
    offset 1886236
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x00000016
 reserved1 0
 reserved2 0
Section
  sectname __gcc_except_tab
   segname __TEXT
      addr 0x00000000001cc820
      size 0x000000000000e7b0
    offset 1886240
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __const
   segname __TEXT
      addr 0x00000000001dafd0
      size 0x000000000001653d
    offset 1945552
     align 2^4 (16)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __cstring
   segname __TEXT
      addr 0x00000000001f1510
      size 0x000000000000cd6e
    offset 2037008
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000002
 reserved1 0
 reserved2 0
Section
  sectname __unwind_info
   segname __TEXT
      addr 0x00000000001fe280
      size 0x0000000000005ca4
    offset 2089600
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __eh_frame
   segname __TEXT
      addr 0x0000000000203f28
      size 0x00000000000000d8
    offset 2113320
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Load command 1
      cmd LC_SEGMENT_64
  cmdsize 232
  segname __DATA_CONST
   vmaddr 0x0000000000204000
   vmsize 0x0000000000010000
  fileoff 2113536
 filesize 65536
  maxprot 0x00000003
 initprot 0x00000003
   nsects 2
    flags 0x10
Section
  sectname __got
   segname __DATA_CONST
      addr 0x0000000000204000
      size 0x0000000000001378
    offset 2113536
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000006
 reserved1 231 (index into indirect symbol table)
 reserved2 0
Section
  sectname __const
   segname __DATA_CONST
      addr 0x0000000000205378
      size 0x000000000000ba08
    offset 2118520
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Load command 2
      cmd LC_SEGMENT_64
  cmdsize 632
  segname __DATA
   vmaddr 0x0000000000214000
   vmsize 0x0000000000004000
  fileoff 2179072
 filesize 16384
  maxprot 0x00000003
 initprot 0x00000003
   nsects 7
    flags 0x0
Section
  sectname __thread_vars
   segname __DATA
      addr 0x0000000000214000
      size 0x0000000000000168
    offset 2179072
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000013
 reserved1 0
 reserved2 0
Section
  sectname __data
   segname __DATA
      addr 0x0000000000214168
      size 0x000000000000004c
    offset 2179432
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __thread_ptrs
   segname __DATA
      addr 0x00000000002141b8
      size 0x0000000000000010
    offset 2179512
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000014
 reserved1 854 (index into indirect symbol table)
 reserved2 0
Section
  sectname __thread_data
   segname __DATA
      addr 0x00000000002141c8
      size 0x0000000000000250
    offset 2179528
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000011
 reserved1 0
 reserved2 0
Section
  sectname __thread_bss
   segname __DATA
      addr 0x0000000000214418
      size 0x00000000000001d0
    offset 0
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000012
 reserved1 0
 reserved2 0
Section
  sectname __common
   segname __DATA
      addr 0x00000000002145f0
      size 0x0000000000000060
    offset 0
     align 2^4 (16)
    reloff 0
    nreloc 0
     flags 0x00000001
 reserved1 0
 reserved2 0
Section
  sectname __bss
   segname __DATA
      addr 0x0000000000214650
      size 0x0000000000000138
    offset 0
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000001
 reserved1 0
 reserved2 0
Load command 3
      cmd LC_SEGMENT_64
  cmdsize 72
  segname __LINKEDIT
   vmaddr 0x0000000000218000
   vmsize 0x0000000000118000
  fileoff 2195456
 filesize 1142860
  maxprot 0x00000001
 initprot 0x00000001
   nsects 0
    flags 0x0
Load command 4
          cmd LC_ID_DYLIB
      cmdsize 48
         name @rpath/lib_lightgbm.so (offset 24)
   time stamp 1 Thu Jan  1 01:00:01 1970
      current version 0.0.0
compatibility version 0.0.0
Load command 5
      cmd LC_DYLD_CHAINED_FIXUPS
  cmdsize 16
  dataoff 2195456
 datasize 47720
Load command 6
      cmd LC_DYLD_EXPORTS_TRIE
  cmdsize 16
  dataoff 2243176
 datasize 42776
Load command 7
     cmd LC_SYMTAB
 cmdsize 24
  symoff 2293192
   nsyms 8937
  stroff 2439608
 strsize 872688
Load command 8
            cmd LC_DYSYMTAB
        cmdsize 80
      ilocalsym 0
      nlocalsym 7736
     iextdefsym 7736
     nextdefsym 982
      iundefsym 8718
      nundefsym 219
         tocoff 0
           ntoc 0
      modtaboff 0
        nmodtab 0
   extrefsymoff 0
    nextrefsyms 0
 indirectsymoff 2436184
  nindirectsyms 856
      extreloff 0
        nextrel 0
      locreloff 0
        nlocrel 0
Load command 9
     cmd LC_UUID
 cmdsize 24
    uuid 541AECD0-B371-30C6-BF59-E2AEDDB81D6A
Load command 10
      cmd LC_BUILD_VERSION
  cmdsize 32
 platform 1
    minos 12.0
      sdk 12.3
   ntools 1
     tool 3
  version 764.0
Load command 11
      cmd LC_SOURCE_VERSION
  cmdsize 16
  version 0.0
Load command 12
          cmd LC_LOAD_DYLIB
      cmdsize 72
         name /opt/homebrew/opt/libomp/lib/libomp.dylib (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 5.0.0
compatibility version 5.0.0
Load command 13
          cmd LC_LOAD_DYLIB
      cmdsize 48
         name /usr/lib/libc++.1.dylib (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 1300.23.0
compatibility version 1.0.0
Load command 14
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name /usr/lib/libSystem.B.dylib (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 1311.100.3
compatibility version 1.0.0
Load command 15
      cmd LC_FUNCTION_STARTS
  cmdsize 16
  dataoff 2285952
 datasize 7240
Load command 16
      cmd LC_DATA_IN_CODE
  cmdsize 16
  dataoff 2293192
 datasize 0
Load command 17
      cmd LC_CODE_SIGNATURE
  cmdsize 16
  dataoff 3312304
 datasize 26012

and

~/software/LightGBM master ❯ otool -l /Users/michelsen/.julia/packages/LightGBM/A7zVd/src/lib_lightgbm.dylib
/Users/michelsen/.julia/packages/LightGBM/A7zVd/src/lib_lightgbm.dylib:
Load command 0
      cmd LC_SEGMENT_64
  cmdsize 632
  segname __TEXT
   vmaddr 0x0000000000000000
   vmsize 0x0000000000251000
  fileoff 0
 filesize 2428928
  maxprot 0x00000007
 initprot 0x00000005
   nsects 7
    flags 0x0
Section
  sectname __text
   segname __TEXT
      addr 0x0000000000001b60
      size 0x000000000020c409
    offset 7008
     align 2^4 (16)
    reloff 0
    nreloc 0
     flags 0x80000400
 reserved1 0
 reserved2 0
Section
  sectname __stubs
   segname __TEXT
      addr 0x000000000020df6a
      size 0x0000000000000564
    offset 2154346
     align 2^1 (2)
    reloff 0
    nreloc 0
     flags 0x80000408
 reserved1 0 (index into indirect symbol table)
 reserved2 6 (size of stubs)
Section
  sectname __stub_helper
   segname __TEXT
      addr 0x000000000020e4d0
      size 0x00000000000006dc
    offset 2155728
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x80000400
 reserved1 0
 reserved2 0
Section
  sectname __gcc_except_tab
   segname __TEXT
      addr 0x000000000020ebac
      size 0x000000000001d87c
    offset 2157484
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __const
   segname __TEXT
      addr 0x000000000022c430
      size 0x0000000000014949
    offset 2278448
     align 2^4 (16)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __cstring
   segname __TEXT
      addr 0x0000000000240d79
      size 0x000000000000b04d
    offset 2362745
     align 2^0 (1)
    reloff 0
    nreloc 0
     flags 0x00000002
 reserved1 0
 reserved2 0
Section
  sectname __unwind_info
   segname __TEXT
      addr 0x000000000024bdc8
      size 0x0000000000005234
    offset 2407880
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Load command 1
      cmd LC_SEGMENT_64
  cmdsize 1032
  segname __DATA
   vmaddr 0x0000000000251000
   vmsize 0x000000000000f000
  fileoff 2428928
 filesize 61440
  maxprot 0x00000007
 initprot 0x00000003
   nsects 12
    flags 0x0
Section
  sectname __nl_symbol_ptr
   segname __DATA
      addr 0x0000000000251000
      size 0x0000000000000010
    offset 2428928
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000006
 reserved1 230 (index into indirect symbol table)
 reserved2 0
Section
  sectname __got
   segname __DATA
      addr 0x0000000000251010
      size 0x0000000000001818
    offset 2428944
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000006
 reserved1 232 (index into indirect symbol table)
 reserved2 0
Section
  sectname __la_symbol_ptr
   segname __DATA
      addr 0x0000000000252828
      size 0x0000000000000730
    offset 2435112
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000007
 reserved1 1003 (index into indirect symbol table)
 reserved2 0
Section
  sectname __mod_init_func
   segname __DATA
      addr 0x0000000000252f58
      size 0x00000000000000d8
    offset 2436952
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000009
 reserved1 0
 reserved2 0
Section
  sectname __const
   segname __DATA
      addr 0x0000000000253030
      size 0x00000000000017d8
    offset 2437168
     align 2^4 (16)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __thread_vars
   segname __DATA
      addr 0x0000000000254808
      size 0x0000000000000168
    offset 2443272
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000013
 reserved1 0
 reserved2 0
Section
  sectname __data
   segname __DATA
      addr 0x0000000000254970
      size 0x000000000000a8e8
    offset 2443632
     align 2^4 (16)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __thread_ptrs
   segname __DATA
      addr 0x000000000025f258
      size 0x0000000000000010
    offset 2486872
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000014
 reserved1 0 (index into indirect symbol table)
 reserved2 0
Section
  sectname __thread_data
   segname __DATA
      addr 0x000000000025f270
      size 0x0000000000000254
    offset 2486896
     align 2^4 (16)
    reloff 0
    nreloc 0
     flags 0x00000011
 reserved1 0
 reserved2 0
Section
  sectname __thread_bss
   segname __DATA
      addr 0x000000000025f4d0
      size 0x00000000000001e0
    offset 0
     align 2^4 (16)
    reloff 0
    nreloc 0
     flags 0x00000012
 reserved1 0
 reserved2 0
Section
  sectname __common
   segname __DATA
      addr 0x000000000025f6b0
      size 0x0000000000000060
    offset 0
     align 2^4 (16)
    reloff 0
    nreloc 0
     flags 0x00000001
 reserved1 0
 reserved2 0
Section
  sectname __bss
   segname __DATA
      addr 0x000000000025f710
      size 0x0000000000000108
    offset 0
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000001
 reserved1 0
 reserved2 0
Load command 2
      cmd LC_SEGMENT_64
  cmdsize 72
  segname __LINKEDIT
   vmaddr 0x0000000000260000
   vmsize 0x0000000000138000
  fileoff 2490368
 filesize 1275028
  maxprot 0x00000007
 initprot 0x00000001
   nsects 0
    flags 0x0
Load command 3
          cmd LC_ID_DYLIB
      cmdsize 48
         name @rpath/lib_lightgbm.so (offset 24)
   time stamp 1 Thu Jan  1 01:00:01 1970
      current version 0.0.0
compatibility version 0.0.0
Load command 4
            cmd LC_DYLD_INFO_ONLY
        cmdsize 48
     rebase_off 2490368
    rebase_size 2048
       bind_off 2492416
      bind_size 3496
  weak_bind_off 2495912
 weak_bind_size 165056
  lazy_bind_off 2660968
 lazy_bind_size 6360
     export_off 2667328
    export_size 94216
Load command 5
     cmd LC_SYMTAB
 cmdsize 24
  symoff 2768336
   nsyms 8302
  stroff 2906100
 strsize 859296
Load command 6
            cmd LC_DYSYMTAB
        cmdsize 80
      ilocalsym 0
      nlocalsym 6107
     iextdefsym 6107
     nextdefsym 1960
      iundefsym 8067
      nundefsym 235
         tocoff 0
           ntoc 0
      modtaboff 0
        nmodtab 0
   extrefsymoff 0
    nextrefsyms 0
 indirectsymoff 2901168
  nindirectsyms 1233
      extreloff 0
        nextrel 0
      locreloff 0
        nlocrel 0
Load command 7
     cmd LC_UUID
 cmdsize 24
    uuid 2F166C75-AF1D-3527-BADE-460A476469D9
Load command 8
      cmd LC_VERSION_MIN_MACOSX
  cmdsize 16
  version 10.13
      sdk 10.13
Load command 9
      cmd LC_SOURCE_VERSION
  cmdsize 16
  version 0.0
Load command 10
          cmd LC_LOAD_DYLIB
      cmdsize 64
         name /usr/local/opt/libomp/lib/libomp.dylib (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 5.0.0
compatibility version 5.0.0
Load command 11
          cmd LC_LOAD_DYLIB
      cmdsize 48
         name /usr/lib/libc++.1.dylib (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 400.9.0
compatibility version 1.0.0
Load command 12
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name /usr/lib/libSystem.B.dylib (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 1252.50.4
compatibility version 1.0.0
Load command 13
      cmd LC_FUNCTION_STARTS
  cmdsize 16
  dataoff 2761544
 datasize 6560
Load command 14
      cmd LC_DATA_IN_CODE
  cmdsize 16
  dataoff 2768104
 datasize 232

By the way, thanks a lot for all your help!

yaxxie commented 2 years ago

So, I'm not super familiar with otool output but the fact that each of those points at different locations is definitely concerning/interesting

Show me the outputs for:

ls /usr/local/opt/libomp/lib/libomp.dylib
ls /opt/homebrew/opt/libomp/lib/libomp.dylib

ChristianMichelsen commented 2 years ago

~/software/LightGBM master ❯ ls /usr/local/opt/libomp/lib/libomp.dylib
"/usr/local/opt/libomp/lib/libomp.dylib": No such file or directory (os error 2)

~/software/LightGBM master ❯ ls /opt/homebrew/opt/libomp/lib/libomp.dylib
/opt/homebrew/opt/libomp/lib/libomp.dylib

I'm assuming the result of the first command is also a bit concerning?

yaxxie commented 2 years ago

Yeah so something weird for me is that the so file (I assume this is one you compiled?) links to a file present in your system but the dylib one (which I assume is the supplied one) seems to be with a fixed link to a linkable binary, which isn't present on your system.

My hypothesis is that if you move the dylib away, and move the so to have a dylib extension, it will work. But how your whole system came to be in this state, I can't help so easily with :sweat_smile:

If you could try my suggestion and report back, I'd appreciate it.

The reason why you need to move the so to have dylib extension is just an artifact of how dlopen works, if you give a libname without an extension, it will look for the lib with the current platform specific extension (in your case, dylib) https://github.com/IQVIA-ML/LightGBM.jl/blob/47d8de6ce888b20ded1a471e00ae3fd1417bd2a6/src/LightGBM.jl#L42

ChristianMichelsen commented 2 years ago

Okay, so I went into my /Users/michelsen/.julia/packages/LightGBM/A7zVd/src/ directory.

I deleted the original lib_lightgbm.dylib and changed the extension of lib_lightgbm.so to lib_lightgbm.dylib. I hope this is what you meant?

However, I still get:

ERROR: InitError: LightGBM.LibraryNotFoundError("lib_lightgbm not found. Please ensure this library is either in system dirs or the dedicated paths: [\"/Users/michelsen/.julia/packages/LightGBM/A7zVd/src\"]")

when I try to import LightGBM in a new Julia session. I might just start over, deleting LightGBM both from Homebrew and my own compiled version (and also delete LightGBM.jl). So, with a new fresh start, what would the approach be?

yaxxie commented 2 years ago

It is supposed to work out of the box with just brew install libomp and ] add LightGBM. This is effectively what the CI tests when it runs the macOS version

Perhaps you can share some relevant details of your mac such as:

OS version
Is it M1
Any unusual installation things you might have done
Other things you think might be relevant

Then, as a sanity test, do the following:

pip install virtualenv
virtualenv lgbm
source lgbm/bin/activate
pip install --upgrade setuptools wheel pip
pip install lightgbm

and then try to use lightgbm via python in that virtualenv. This will effectively test whether the standard pypi distributed lightgbm works properly on your machine or not.

ChristianMichelsen commented 2 years ago

Oh, I am sorry about not mentioning this before!

I am running MacOS Monterey (12.4) on an Apple M1 Pro. I have not done anything unusual, at least not intentionally. But yeah, it is an M1, and I've previously run into problems with the native Julia version, so right now I am running 1.7.3 in rosetta mode.

Looking back, I have previously had some trouble with make and homebrew, where I had to run:

export CPATH=/opt/homebrew/include
export LIBRARY_PATH=/opt/homebrew/lib

before make. I do not know if this is relevant or not.

I tried to run your sanity check and it worked without any problems:

~/work ❯ python
Python 3.10.4 (main, Apr 26 2022, 19:36:29) [Clang 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import lightgbm as lgb
>>>

(the above is running in the new lgbm environment.

yaxxie commented 2 years ago

Ok so the next step is to try to find the binary associated with the python installation and copy it to the julia installation

Something like

find lgbm | grep lib_lightgbm
cp <found/file> /Users/michelsen/.julia/packages/LightGBM/A7zVd/src/lib_lightgbm.dylib

if it has an so extension, move it so that it has a dylib extension

yaxxie commented 2 years ago

The other obvious thing to try is

export LIBRARY_PATH=/opt/homebrew/lib

before trying to run the julia process to test LGBM.

yaxxie commented 2 years ago

One last thing, try to run some LGBM code in the python. I didn't think about this until now but the library loading may be deferred

yaxxie commented 2 years ago

@ChristianMichelsen any updates?

FatemehTahavori commented 2 years ago

I have the same issue after changing the laptop, @ChristianMichelsen did you find any solution for this issue?

mattarnoldbio commented 2 years ago

I am having the same problem on Pop!_OS (basically the same as Ubuntu). OpenMP appears to be working fine (I have tested with a gcc -fopenmp MWE). Additionally, the lib_lightgbm.so file is compiled in my cloned LightGBM repo (saved to /usr/local) and symlinked to my /usr/local/lib directory). The output of sudo ldconfig -v shows a line for lib_lightgbm.so at the expected location. Even after running export LIBRARY_PATH=/usr/local/lib, opening a Julia terminal and issuing using LightGBM still produces the same message [ Info: lib_lightgbm not found in system dirs, trying fallback). Any help would be greatly appreciated!

yaxxie commented 2 years ago

@mattarnoldbio can you try with LD_LIBRARY_PATH

yaxxie commented 2 years ago

@FatemehTahavori did you switch to a mac m1?

FatemehTahavori commented 2 years ago

I think the issue is with libpath = Libdl.find_library(library_name, custom_paths), I tested with library_name and custom_paths and still was empty. Yes I should use M1 for now but I had the same issue in colab: https://colab.research.google.com/drive/199DpOU113G7YcYfXQlXhsLs3Bs91Rqbr?usp=sharing.

FatemehTahavori commented 2 years ago

I managed to sort out this issue (I am M1 mac user). I am not sure what actually fixed the issue. I would like one of you try out these steps:

cd ~
rm -rf .julia
delete any project and Manifest toml in this location
(@v1.7) pkg> add LightGBM#Fix_lgbm_libNotFound If the above didn't solve your issue:
first follow this instructions: build-threadless-version-not-recommended https://lightgbm.readthedocs.io/en/latest/Installation-Guide.html#build-threadless-version-not-recommended
- redo the solution one If the second solution didn't work
- arch -arm64 /usr/local/bin/brew install cmake libomp
- redo the step 2 and 1 please let me know which step works for you.

abeot commented 2 years ago

Hi, I also have the same issue and tried the steps @fatemehTahavori recommended. However, it does not. I think the error might occur when LightGBM is installed without libomp, because that is what I did.

yaxxie commented 2 years ago

Hi @albertc100

However, it does not.

Sorry I didn't understand, did you mean to say it does not work or the issue does not occur any longer after trying the steps?

I believe I understand the root cause, would you be willing to try some alternative steps?

abeot commented 2 years ago

Hello @yaxxie, Sorry for the typo, I was trying to say the problem did not go away. If there are some alternative steps, then I would be willing to take them.

yaxxie commented 2 years ago

There are two options here if I understood the root cause correctly: Linking libraries across architectures cannot work (sounds obvious when written down). The entire chain needs to be for one architecture or another (Intel or ARM)

Julia 1.7 ships Intel only Julia 1.8 RCs are shipping with experimental ARM builds (link)

If you want to go Intel builds, or don't want to use 1.8 yet, you need to make sure you have rosetta enabled brew installed. This should help: https://www.wisdomgeek.com/development/installing-intel-based-packages-using-homebrew-on-the-m1-mac/

Then, if you're using Intel julia, you should arch -x86_64 brew install libomp arch -x86_64 brew install lightgbm. This will get the libs for you. I'm not sure exactly where they will be, but you should try looking in /opt/homebrew/lib. Then, you can try running julia by doing DYLD_LIBRARY_PATH=<the_location> julia and it should work.

It is basically the same if you want to use the ARM build, but instead you're restricted to julia 1.8 (experimental build) and instead of doing arch -x86_64 brew install simply do brew install.

The critical bit here really is making sure your installed libs architecture matches the choice of julia binary architecture.

If you're compiling own lightgbm from source you'll need to take additional steps to find out / know what architecture you're compiling for. N.B. the bundled binary is Intel.

I hope this helps. If you have any specific issues please give as much information as possible, and I'll try to help.

abeot commented 2 years ago

Thanks! For some reason, I still get an error even though the file lib_lightgbm.dylib is in the /opt/homebrew/lib directory. Furthermore, when I run echo $DYLD_LIBRARY_PATH I get /opt/homebrew/lib, but the InitError is still thrown.

EDIT: I fixed it by setting the path to /usr/local/homebrew/opt/lightgbm/lib. This may have worked because this was the Intel installed by the Rosetta homebrew.

Thanks again!

yaxxie commented 2 years ago

Thanks for taking the time to try this out. Could you just confirm then that you were using rosetta julia and did the brew install with arch -x86_64?

abeot commented 2 years ago

Yes, the brew did install. I checked the both directories and found Homebrew.

kainkad commented 4 months ago

Closing as completed in v0.7.0

IQVIA-ML / LightGBM.jl

InitError: LightGBM.LibraryNotFoundError and ArgumentError: NULL library handle #122