search

kbarbary / Dierckx.jl

Julia package for 1-d and 2-d splines
Other
157 stars 30 forks source link

building error for v0.4.0 #53

Closed EthanAnderes closed 6 years ago

EthanAnderes commented 6 years ago

Really happy to see work on Dierckx for Julia v0.7 and v1.0 ... but getting build errors. Here is what I get after deleting ~/.julia for fresh install. 

(v0.7) pkg> add Dierckx
   Cloning default registries into /Users/ethananderes/.julia/registries
   Cloning registry General from "https://github.com/JuliaRegistries/General.git"
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
 Resolving package versions...
 Installed Dierckx ──────── v0.4.0
 Installed BinaryProvider ─ v0.4.0
  Updating `~/.julia/environments/v0.7/Project.toml`
  [39dd38d3] + Dierckx v0.4.0
  Updating `~/.julia/environments/v0.7/Manifest.toml`
  [b99e7846] + BinaryProvider v0.4.0
  [39dd38d3] + Dierckx v0.4.0
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [8ba89e20] + Distributed
  [b77e0a4c] + InteractiveUtils
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [44cfe95a] + Pkg
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  Building Dierckx → `~/.julia/packages/Dierckx/5DgyM/deps/build.log`
┌ Error: Error building `Dierckx`:
│ [ Info: Downloading https://github.com/kbarbary/DierckxBuilder/releases/download/v1.0.0/libddierckx.v1.0.0.x86_64-apple-darwin14.tar.gz to /Users/ethananderes/.julia/packages/Dierckx/5DgyM/deps/usr/downloads/libddierckx.v1.0.0.x86_64-apple-darwin14.tar.gz...
│ ERROR: LoadError: LibraryProduct(nothing, ["libddierckx"], :libddierckx, "Prefix(/Users/ethananderes/.julia/packages/Dierckx/5DgyM/deps/usr)") is not satisfied, cannot generate deps.jl!
│ Stacktrace:
│  [1] error(::String) at ./error.jl:33
│  [2] #write_deps_file#134(::Bool, ::Function, ::String, ::Array{LibraryProduct,1}) at /Users/ethananderes/.julia/packages/BinaryProvider/ZbFxn/src/Products.jl:393
│  [3] write_deps_file(::String, ::Array{LibraryProduct,1}) at /Users/ethananderes/.julia/packages/BinaryProvider/ZbFxn/src/Products.jl:380
│  [4] top-level scope at none:0
│  [5] include at ./boot.jl:317 [inlined]
│  [6] include_relative(::Module, ::String) at ./loading.jl:1038
│  [7] include(::Module, ::String) at ./sysimg.jl:29
│  [8] include(::String) at ./client.jl:398
│  [9] top-level scope at none:0
│ in expression starting at /Users/ethananderes/.julia/packages/Dierckx/5DgyM/deps/build.jl:38
[03:21:03] ######################################################################## 100.0%
└ @ Pkg.Operations ~/Software/julia7/usr/share/julia/stdlib/v0.7/Pkg/src/Operations.jl:1068

julia> versioninfo()
Julia Version 0.7.1-pre.0
Commit 36cddc1006 (2018-08-09 00:19 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin17.7.0)
  CPU: Intel(R) Core(TM) i7-8559U CPU @ 2.70GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, skylake)
Environment:
  JULIA_FFTW_PROVIDER = MKL

This reminds me of the problems with Arpack for users who build Julia from source issue 5. Indeed the error message I get is very similar. Just thought I would mention in case the error is related.

kbarbary commented 6 years ago

Thanks for reporting. Did the binary download and unpack correctly to /Users/ethananderes/.julia/packages/Dierckx/5DgyM/deps/usr/lib?

If it is there, what happens when you try to dlopen it in Julia? e.g. Libdl.dlopen("./libddierckx.dylib"). Trying to distinguish if this is a linking problem or something more mundane.

EthanAnderes commented 6 years ago

Thanks for looking into it so quickly. Yes, it did appear to download (not sure how to check if it unpacked correctly).

Here is what I get when I run the command you suggested ...

shell> ls
libddierckx.dylib

shell> pwd
/Users/ethananderes/.julia/packages/Dierckx/5DgyM/deps/usr/lib

julia> using Libdl

julia> Libdl.dlopen("./libddierckx.dylib")
ERROR: could not load library "./libddierckx.dylib"
dlopen(./libddierckx.dylib, 1): Library not loaded: @rpath/libgfortran.4.dylib
  Referenced from: /Users/ethananderes/.julia/packages/Dierckx/5DgyM/deps/usr/lib/libddierckx.dylib
  Reason: image not found
Stacktrace:
 [1] dlopen(::String, ::UInt32) at /Users/ethananderes/Software/julia7/usr/share/julia/stdlib/v0.7/Libdl/src/Libdl.jl:97 (repeats 2 times)
 [2] top-level scope at none:0

Sorry I'm much help with dlopen. It's a bit above my knowledge level. But I would be happy to try anything you would suggest.

EthanAnderes commented 6 years ago

One other piece of info that I'm not sure is relevant ... I used gcc 8.2 to compile Julia. Looking through the comments in Arpack.jl there was some suggestion that part of the problem was gcc-7 compatibility.

Anyway, thought I would toss it out there to help try and reproduce the problem.

kbarbary commented 6 years ago

Thanks, that's helpful. I think the problem might be that libgfortran isn't being found. I think statically linking it during compilation would fix this. I'm not really an expert though, so I'm going to wait til I have a chance to ask around on slack or BinaryBuilder abound it before doing that. Hopefully this week.

If you need an interim fix, you could try to compile from source (kbarbary/DierckxBuilder ; run make in src/) and then manually copy the created libddierckx.dylib file to Users/ethananderes/.julia/packages/Dierckx/5DgyM/deps/usr/lib

EthanAnderes commented 6 years ago

Ok, so I tried running make in a cloned version of DierckxBuilder/src/ddierckx. It worked fine but only produced libddierckx.so and not libddierckx.dylib. Not sure if this is a clue to where the problem is, but I thought I would mention it.

kbarbary commented 6 years ago

The suffix name is unrelated (I actually use this command instead of make when building, so the suffix is dylib for mac. You can just manually rename the file and it hopefully will work.

I think you were right that it is a version discrepancy due to building from source with a more-recent GCC. Here's the identical issue you're having along with a reply explaining the problem (and current lack of solution): https://github.com/JuliaMath/OpenspecfunBuilder/issues/9

And here I thought distributing binaries would solve all my problems...

EthanAnderes commented 6 years ago

Ok good. Happy to know the problem is isolated. I'll follow that issue to see what solutions people come up with. In the meantime I've got v0.3 pinned in the package system which runs on Julia v0.7 so don't need any other workaround. Thanks a ton and let me know when/if there is any help you need.

jdtuck commented 6 years ago

I am having the same issue when julia is compiled from source

EthanAnderes commented 6 years ago

Here is a work around that I found work on my machine (macOS). I first installed gcc7 with homebrew (can't recall the exact command I used but it was something like brew install gcc@7). Then I added this to my .bash_profile. 

alias jm="DYLD_LIBRARY_PATH=/usr/local/opt/gcc@7/lib/gcc/7 /Users/ethananderes/Software/juliaMaste/Julia"

In a fresh bash terminal launch Julia with jm. Now building Dierckx works fine.

davorh commented 6 years ago

Any solution for Julia 1.0? I'm getting an error on Debian 8.

Building Dierckx → ~/.julia/packages/Dierckx/16oZP/deps/build.log ┌ Error: Error building Dierckx: │ ┌ Warning: platform_key() is deprecated, use platform_key_abi() from now on │ │ caller = ip:0x0 │ └ @ Core :-1 │ ERROR: LoadError: LibraryProduct(nothing, ["libddierckx"], :libddierckx, "Prefix(~/.julia/packages/Dierckx/16oZP/deps/usr)") is not satisfied, cannot generate deps.jl! │ Stacktrace: │ [1] error(::String) at ./error.jl:33 │ [2] #write_deps_file#152(::Bool, ::Function, ::String, ::Array{LibraryProduct,1}) at ~/.julia/packages/BinaryProvider/cVlaj/src/Products.jl:408 │ [3] write_deps_file(::String, ::Array{LibraryProduct,1}) at ~/.julia/packages/BinaryProvider/cVlaj/src/Products.jl:395 │ [4] top-level scope at none:0 │ [5] include at ./boot.jl:317 [inlined] │ [6] include_relative(::Module, ::String) at ./loading.jl:1038 │ [7] include(::Module, ::String) at ./sysimg.jl:29 │ [8] include(::String) at ./client.jl:388 │ [9] top-level scope at none:0 │ in expression starting at ~/.julia/packages/Dierckx/16oZP/deps/build.jl:38 └ @ Pkg.Operations /opt/julia/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1068

kbarbary commented 6 years ago

Seems like this might be a different problem. Did you build Julia from source or use the pre-built binaries? Is this a standard architecture machine? (I'm asking because the libdierckx binaries are currently only build for x86 32-bit and 64-bit bit.)

davorh commented 6 years ago

Standard Debian 8 x86 64-bit, Intel Xeon, gcc etc., Julia compiled from the source. Installed bunch of packages without any hickup :) Used "add Dierckx" as instructed.

kbarbary commented 6 years ago

Looks like BinaryBuilder supports multi-versioning of gcc now (as of yesterday), so I'll try updating DierckxBuilder to use that. I might not have a chance to do it right away though... there are some short term fixes earlier in this thread if you need it now.

davorh commented 6 years ago

Tnx and thx for the great work! I'll be patient :)

EthanAnderes commented 6 years ago

Just for informational purposes .... I also tried a fresh build of Julia (master branch, today) and new BinaryBuilder.jl ... got similar errors as @davorh . Luckily the work around for MacOS still works for me:)

Would also like to second @davorh 's comment: great work ... worth the wait.

kbarbary commented 6 years ago

I tagged v0.4.1 that includes gcc-multiversioned binaries, designed to fix this issue. If anyone who was having this issue can try it out and report back, that would be great!

EthanAnderes commented 6 years ago

hmmm, I'm still getting 

┌ Error: Error building `Dierckx`: 
│ [ Info: Downloading https://github.com/kbarbary/DierckxBuilder/releases/download/v1.0.0-1/libddierckx.v1.0.0.x86_64-apple-darwin14-gcc8.tar.gz to /Users/ethananderes/.julia/packages/Dierckx/9SY4A/deps/usr/downloads/libddierckx.v1.0.0.x86_64-apple-darwin14-gcc8.tar.gz...
│ ERROR: LoadError: LibraryProduct(nothing, ["libddierckx"], :libddierckx, "Prefix(/Users/ethananderes/.julia/packages/Dierckx/9SY4A/deps/usr)") is not satisfied, cannot generate deps.jl!
│ Stacktrace:
│  [1] error(::String) at ./error.jl:33
│  [2] #write_deps_file#152(::Bool, ::Function, ::String, ::Array{LibraryProduct,1}) at /Users/ethananderes/.julia/packages/BinaryProvider/cVlaj/src/Products.jl:408
│  [3] (::getfield(BinaryProvider, Symbol("#kw##write_deps_file")))(::NamedTuple{(:verbose,),Tuple{Bool}}, ::typeof(write_deps_file), ::String, ::Array{LibraryProduct,1}) at ./none:0
│  [4] top-level scope at none:0
│  [5] include at ./boot.jl:317 [inlined]
│  [6] include_relative(::Module, ::String) at ./loading.jl:1038
│  [7] include(::Module, ::String) at ./sysimg.jl:29
│  [8] include(::String) at ./client.jl:388
│  [9] top-level scope at none:0
│ in expression starting at /Users/ethananderes/.julia/packages/Dierckx/9SY4A/deps/build.jl:55
[03:55:01] ######################################################################## 100.0% 
└ @ Pkg.Operations ~/Software/juliaMaster/usr/share/julia/stdlib/v1.1/Pkg/src/Operations.jl:1069

with macOS Mojave and ... 

julia> versioninfo()
Julia Version 1.1.0-DEV.429
Commit 361c5e76b5 (2018-10-08 07:52 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin18.0.0)
  CPU: Intel(R) Core(TM) i7-8559U CPU @ 2.70GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, skylake)
Environment:
  JULIA_FFTW_PROVIDER = MKL
EthanAnderes commented 6 years ago

Ok, this is weird. Even my workaround doesn't work for v0.4.1.

Here is what I get when I use DYLD_LIBRARY_PATH=/usr/local/opt/gcc@7/lib/gcc/7 /Users/ethananderes/Software/juliaMaster/julia -O3

(v1.1) pkg> build Dierckx
  Building Dierckx → `~/.julia/packages/Dierckx/9SY4A/deps/build.log`
┌ Error: Error building `Dierckx`: 
│ ERROR: LoadError: LibraryProduct(nothing, ["libddierckx"], :libddierckx, "Prefix(/Users/ethananderes/.julia/packages/Dierckx/9SY4A/deps/usr)") is not satisfied, cannot generate deps.jl!
│ Stacktrace:
│  [1] error(::String) at ./error.jl:33
│  [2] #write_deps_file#152(::Bool, ::Function, ::String, ::Array{LibraryProduct,1}) at /Users/ethananderes/.julia/packages/BinaryProvider/cVlaj/src/Products.jl:408
│  [3] (::getfield(BinaryProvider, Symbol("#kw##write_deps_file")))(::NamedTuple{(:verbose,),Tuple{Bool}}, ::typeof(write_deps_file), ::String, ::Array{LibraryProduct,1}) at ./none:0
│  [4] top-level scope at none:0
│  [5] include at ./boot.jl:317 [inlined]
│  [6] include_relative(::Module, ::String) at ./loading.jl:1038
│  [7] include(::Module, ::String) at ./sysimg.jl:29
│  [8] include(::String) at ./client.jl:388
│  [9] top-level scope at none:0
│ in expression starting at /Users/ethananderes/.julia/packages/Dierckx/9SY4A/deps/build.jl:55
└ @ Pkg.Operations ~/Software/juliaMaster/usr/share/julia/stdlib/v1.1/Pkg/src/Operations.jl:1069

(v1.1) pkg> add Dierckx@v0.4.0
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
 Resolving package versions...
 Installed Dierckx ─ v0.4.0
  Updating `~/.julia/environments/v1.1/Project.toml`
  [39dd38d3] ↓ Dierckx v0.4.1 ⇒ v0.4.0
  Updating `~/.julia/environments/v1.1/Manifest.toml`
  [39dd38d3] ↓ Dierckx v0.4.1 ⇒ v0.4.0
  Building Dierckx → `~/.julia/packages/Dierckx/5DgyM/deps/build.log`

So, in summary v0.4.1 doesn't work for me with or without setting DYLD_LIBRARY_PATH to gcc7... but 0.4.0 works with DYLD_LIBRARY_PATH set to gcc7 but not gcc8. Hope this helps track down the issue.

kbarbary commented 6 years ago

Thanks for checking. I'm not quite sure what's going on with v0.4.1 here. It looks like BInaryProvider is detecting that your compiler abi is gcc8 and is downloading that binary. Did you actually build Julia with gcc8 or gcc7? (which libgfortran version do you have under the julia build?)

Can you check that the libddierckx library exists again and try to open it with Libdl.dlopen as in https://github.com/kbarbary/Dierckx.jl/issues/53#issuecomment-412589130?

EthanAnderes commented 6 years ago

Here is what I get from the directory that appears to be generated by Dierckx v0.4.1 (using Julia compiled with gcc8)

julia> Libdl.dlopen("./libddierckx.dylib")
ERROR: could not load library "./libddierckx.dylib"
dlopen(./libddierckx.dylib, 1): Library not loaded: @rpath/libgfortran.5.dylib
  Referenced from: /Users/ethananderes/.julia/packages/Dierckx/9SY4A/deps/usr/lib/libddierckx.dylib
  Reason: image not found
Stacktrace:
 [1] #dlopen#3(::Bool, ::Function, ::String, ::UInt32) at /Users/ethananderes/Software/juliaMaster/usr/share/julia/stdlib/v1.1/Libdl/src/Libdl.jl:109
 [2] dlopen at /Users/ethananderes/Software/juliaMaster/usr/share/julia/stdlib/v1.1/Libdl/src/Libdl.jl:109 [inlined] (repeats 2 times)
 [3] top-level scope at none:0

julia> pwd()
"/Users/ethananderes/.julia/packages/Dierckx/9SY4A/deps/usr/lib"

I get the above message with or without setting DYLD_LIBRARY_PATH=/usr/local/opt/gcc@7/lib/gcc/7.

In contrast, here is what I get from the directory that appears to be generated by Dierckx v0.4.0 (without setting `DYLD_LIBRARY_PATH=/usr/local/opt/gcc@7/lib/gcc/7)

julia> Libdl.dlopen("./libddierckx.dylib")
ERROR: could not load library "./libddierckx.dylib"
dlopen(./libddierckx.dylib, 1): Library not loaded: @rpath/libgfortran.4.dylib
  Referenced from: /Users/ethananderes/.julia/packages/Dierckx/5DgyM/deps/usr/lib/libddierckx.dylib
  Reason: image not found
Stacktrace:
 [1] #dlopen#3(::Bool, ::Function, ::String, ::UInt32) at /Users/ethananderes/Software/juliaMaster/usr/share/julia/stdlib/v1.1/Libdl/src/Libdl.jl:109
 [2] dlopen at /Users/ethananderes/Software/juliaMaster/usr/share/julia/stdlib/v1.1/Libdl/src/Libdl.jl:109 [inlined] (repeats 2 times)
 [3] top-level scope at none:0

julia> pwd()
"/Users/ethananderes/.julia/packages/Dierckx/5DgyM/deps/usr/lib"

... however I get it to work (in the directory generated by Dierckx v0.4.0) in the case I set `DYLD_LIBRARY_PATH=/usr/local/opt/gcc@7/lib/gcc/7 

julia> using Libdl

julia> Libdl.dlopen("./libddierckx.dylib")
Ptr{Nothing} @0x00007fb8cd8263e0

julia> pwd()
"/Users/ethananderes/.julia/packages/Dierckx/5DgyM/deps/usr/lib"

Hope this helps ...

kbarbary commented 6 years ago

So what's going on is that in Dierckx 0.4.1, libddierckx is linked to libgfortran.5.dylib. See this line: 

dlopen(./libddierckx.dylib, 1): Library not loaded: @rpath/libgfortran.5.dylib

libgfortran.5.dylib goes along with gcc8 (yes, they're different numbers: 5 goes with 8). This is why setting DYLD_LIBRARY_PATH=/usr/local/opt/gcc@7/lib/gcc/7 doesn't help in this case, because that directory probably contains libgfortran.4.dylib, not libgfortran.5.dylib.

Two options: (1) try setting DYLD_LIBRARY_PATH to the equivalent gcc8 path, which hopefully contains libgfortran.5.dylib.

(2) I think this comment might explain your situation. You could try following the instructions there.

davorh commented 6 years ago

0.4.1 works perfectly now on Julia 1.0.0 on my system. Tnx!

EthanAnderes commented 6 years ago

Ok, now everything is working if I use 

DYLD_LIBRARY_PATH=/usr/local/opt/gcc/lib/gcc/8 /Users/ethananderes/Software/juliaMaster/julia -O3

It must have something to do with how Julia is setting the lib paths etc ... noticed some activity on this at JuliaLang. Perhaps that will fix having to specify DYLD_LIBRARY_PATH. Anyway, it seems we are all good on your end. Thanks!!

kbarbary commented 6 years ago

Thanks for checking. I'm going to close this for now but anyone should feel free to comment if they're still having related issues.