Building LBFGSB using julia compiled with MKL fails

[Djoop]

I've been successfully using LBFGSB on julia 1.0.2, but recently I tried to compile julia with MKL to compare performances (still julia 1.0.2). When I try to build LBFGSB, I get the following error (I just changed the username):

┌ Warning: Cannot make sense of autodetected libstdc++ ABI version ('3.4.0')
└ @ BinaryProvider ~/.julia/packages/BinaryProvider/4F5Hq/src/PlatformNames.jl:624
┌ Debug: Rejecting cache file /home/USERNAME/.julia/compiled/v1.0/BinaryProvider/ek6VZ.ji due to it containing an invalid cache header
└ @ Base loading.jl:1330
┌ Debug: Recompiling stale cache file /home/USERNAME/.julia/compiled/v1.0/BinaryProvider/ek6VZ.ji for BinaryProvider [b99e7846-7c00-51b0-8f62-c81ae34c0232]
└ @ Base loading.jl:1190
┌ Warning: platform_key() is deprecated, use platform_key_abi() from now on
│   caller = ip:0x0
└ @ Core :-1
ERROR: LoadError: LibraryProduct(nothing, ["liblbfgsb"], :liblbfgsb, "Prefix(/home/USERNAME/.julia/packages/LBFGSB/sLXbo/deps/usr)") is not satisfied, cannot generate deps.jl!
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] #write_deps_file#152(::Bool, ::Function, ::String, ::Array{LibraryProduct,1}) at /home/USERNAME/.julia/packages/BinaryProvider/4F5Hq/src/Products.jl:414
 [3] (::getfield(BinaryProvider, Symbol("#kw##write_deps_file")))(::NamedTuple{(:verbose,),Tuple{Bool}}, ::typeof(write_deps_file), ::String, ::Array{LibraryProduct,1}) at ./none:0
 [4] top-level scope at none:0
 [5] include at ./boot.jl:317 [inlined]
 [6] include_relative(::Module, ::String) at ./loading.jl:1044
 [7] include(::Module, ::String) at ./sysimg.jl:29
 [8] include(::String) at ./client.jl:392
 [9] top-level scope at none:0
in expression starting at /home/USERNAME/.julia/packages/LBFGSB/sLXbo/deps/build.jl:46

I'm not sure to understand what is happening and if the compilation with MKL is really the issue or not, but the library is present and can be opened:

julia> Libdl.open("/home/USERNAME/.julia/packages/LBFGSB/sLXbo/deps/usr/lib/liblbfgsb-3.so")
IOStream(<file /home/USERNAME/.julia/packages/LBFGSB/sLXbo/deps/usr/lib/liblbfgsb-3.so>)

Any hint?

[EDIT] Sorry, posted too quickly, I used open and not dlopen

julia> Libdl.dlopen("/home/USERNAME/.julia/packages/LBFGSB/sLXbo/deps/usr/lib/liblbfgsb-3.so")
ERROR: could not load library "/home/USERNAME/.julia/packages/LBFGSB/sLXbo/deps/usr/lib/liblbfgsb-3.so"
libgfortran.so.4: Ne peut ouvrir le fichier d'objet partagé: Aucun fichier ou dossier de ce type

So I guess this is a more general problem, probably not related directly to LBFGSB or MKL but just to the fact that julia was compiled from source…

[Gnimuc]

So I guess this is a more general problem, probably not related directly to LBFGSB or MKL but just to the fact that julia was compiled from source…

yep. the reason is that the version of libgfortran linked by BB2 is libgfortran.so.3. if I remembered correctly, this has been fixed in the upstream. I'm trying to rebuild LBFGSB binaries using newer BB2.

[Djoop]

OK thanks, I tried with v1.1.0 but I still have the same issue. But when editing LD_LIBRARY_PATH, compilation does work. So I guess there is nothing more to do, just wait for the update?

[Gnimuc]

It seems that simply extending gcc versions is not enough for some platforms: https://travis-ci.org/Gnimuc/L-BFGS-B-Builder/builds/483380231

For debugging locally, I'm updating and downloading BB2 now, so you might need to wait a couple of more hours. :)

[Gnimuc]

hi @Djoop, I've rebuilt LBFGSB binaries for many gcc versions except for powerpc and macOS. you could download and replace the content of your LBFGSB's /deps/build.jl with https://github.com/Gnimuc/L-BFGS-B-Builder/releases/download/v3.0.0-3/build_LBFGSB.v3.0.0.jl

currently, I don't have a Linux machine that has a Julia built from source, so this method is still not tested, please add more comments if things still go bad. for macOS, it looks like I hit this issue: https://github.com/JuliaPackaging/BinaryBuilder.jl/issues/381.

BTW, could you elaborate more about how did you fix the compilation with LD_LIBRARY_PATH?

[Djoop]

Thank you for the quick fix, I works for me on linux by editing /deps/build.jl. Regarding LD_LIBRARY_PATH, I just wanted to check that there was no other error so I added to the library path libgfortran.so.4 (that I had somewhere else on my system), but this is of course not really a solution.

[Gnimuc]

@Djoop I've tagged a new release, you could use it as soon as https://github.com/JuliaLang/METADATA.jl/pull/20831 is merged. Thanks for the feedback.