search

JuliaInterop / libcxxwrap-julia

C++ library for backing CxxWrap.jl
Other
85 stars 43 forks source link

Problem with deploying a julia package using libcxxwrap-julia #115

Closed sunayana closed 1 year ago

sunayana commented 1 year ago

Hello, I have been working on wrapping some C++ code using libcxxwrap-julia library. Here is the link to the C++ code: https://github.com/Cervest/richdem/tree/feature/sc-40452/add_richdem_jll/src The wrapper C++ code sits in : https://github.com/Cervest/richdem/tree/feature/sc-40452/add_richdem_jll/src/jlrichdem I was also able to deploy this on Yggdrasil and have RichDEM_jll.jl package My local development environment works absolutely fine using @wrapmodule calls like e.g., @wrapmodule("/workspaces/richdem/build/lib/libjlrichdem.so", :define_julia_module, Libdl.RTLD_GLOBAL) I have done this following the juliacon 2020 tutorial by @barche .

Now I am at the stage when I change the @wrapmodule call to this: @wrapmodule(RichDEM_jll.libjlrichdem_path, :define_julia_module, Libdl.RTLD_GLOBAL) see this file : https://github.com/Cervest/richdem/blob/feature/sc-40452/add_richdem_jll/wrappers/RichDEM.jl/src/RichDEM.jl

But get an error: Pkg.add(PackageSpec(url="https://github.com/Cervest/richdem.git", subdir="wrappers/RichDEM.jl"))

[ Info: Precompiling RichDEM [94a1c18c-0d95-461b-994e-5c3447d2ff66]
C++ exception while wrapping module common: invalid subtyping in definition of Array2D with supertype AbstractArray
ERROR: LoadError: invalid subtyping in definition of Array2D with supertype AbstractArray
Stacktrace:
 [1] register_julia_module
   @ ~/.julia/packages/CxxWrap/25c5A/src/CxxWrap.jl:393 [inlined]
 [2] readmodule(so_path::String, funcname::Symbol, m::Module, flags::UInt32)
   @ CxxWrap.CxxWrapCore ~/.julia/packages/CxxWrap/25c5A/src/CxxWrap.jl:722
 [3] wrapmodule(so_path::String, funcname::Symbol, m::Module, flags::UInt32)
   @ CxxWrap.CxxWrapCore ~/.julia/packages/CxxWrap/25c5A/src/CxxWrap.jl:726
 [4] include
   @ ./Base.jl:419 [inlined]
 [5] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
   @ Base ./loading.jl:1554
in expression starting at /home/sunayana/.julia/packages/RichDEM/JqNqY/src/RichDEM.jl:1
in expression starting at stdin:1
ERROR: Failed to precompile RichDEM [94a1c18c-0d95-461b-994e-5c3447d2ff66] to /home/sunayana/.julia/compiled/v1.8/RichDEM/jl_o9qcXH.
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
   @ Base ./loading.jl:1707
 [3] compilecache
   @ ./loading.jl:1651 [inlined]
 [4] _require(pkg::Base.PkgId)
   @ Base ./loading.jl:1337
 [5] _require_prelocked(uuidkey::Base.PkgId)
   @ Base ./loading.jl:1200
 [6] macro expansion
   @ ./loading.jl:1180 [inlined]
 [7] macro expansion
   @ ./lock.jl:223 [inlined]
 [8] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1144

Not sure whether this is somehow associated with the C++ compiler. But not sure how to debug this. Any help would be much appreciated. Thanks!

barche commented 1 year ago

This kind of error can happen when mixing JLL binaries with self-compiled binaries. Do you still have the Overrides.toml active for libcxxwrap-julia when testing?

sunayana commented 1 year ago

Thanks @barche, I had Overrides.toml active so I removed it but still get the same error, these are my steps:

sunayana commented 1 year ago

The other thing is that this error will disappear if I copy the self build .so libraries into the expected place:

/root/.julia/artifacts/b717953327b36aae1607d17f1f4a4c7b3cdfdb87/lib/

so not sure what I do wrong in build_tarballs.jl file to generate the binaries correctly.

barche commented 1 year ago

I just tried it myself on my Linux machine and I can reproduce your observations perfectly. No idea yet why this happens.

barche commented 1 year ago

I think this may be because we need to build against the correct libjulia version now, I have created a new Yggdrasil PR at https://github.com/JuliaPackaging/Yggdrasil/pull/6152

Hopefully that will fix it.

sunayana commented 1 year ago

Thanks a lot @barche for looking into this. I am keeping a close watch on the Yggdrasil build, I will test it out once the binaries are available.

sunayana commented 1 year ago

From the GDAL_jll page : https://github.com/JuliaBinaryWrappers/GDAL_jll.jl I see the supported platforms: would something similar : https://github.com/JuliaPackaging/Yggdrasil/blob/master/G/GDAL/build_tarballs.jl#L97 be helpful in this case since GDAL is one of the dependencies.

sunayana commented 1 year ago

Here is another PR I opened around GDAL related changes to build_tarballs.jl https://github.com/JuliaPackaging/Yggdrasil/pull/6154

barche commented 1 year ago

Thanks for the comments. I think there is a lot of overlap between our PRs, so we should decide which one we want to go through. The libcurl error is related to the specific version of the library, which depends on the Julia version since it is a library that is bundled with Julia. I'll ask the Yggdrasil maintainers how to fix this, since I have no idea.

sunayana commented 1 year ago

Would you be fine if I opened a PR in your branch, I would prefer to keep yours active. Thanks again for your help. Also if it helps at the moment I am fine with the build working for version 1.8.x of Julia.

sunayana commented 1 year ago

Just to keep you informed, I closed my PR.

barche commented 1 year ago

I invited you as collaborator to my Yggdrasil fork, so you should be able to edit the PR.

sunayana commented 1 year ago

Thank you @barche . I have figured out why I am having the issue, it is because of the version of GDAL. It works fine in my local testing when I use GDAL version 3.2.2. I am not sure what would be the corresponding GDAL_jll version, I tried with v3.2.1 and it worked. I have opened a PR now with the pinned version : https://github.com/barche/Yggdrasil/pull/1

barche commented 1 year ago

It seems to work mostly, see comment in the PR.

sunayana commented 1 year ago

Thanks @barche . As per your comments I have added OpenMP LLVM package for mac and dropped armv6 from the list of platforms as it was the only one failing in the last commit. Here is the latest PR to your branch : https://github.com/barche/Yggdrasil/pull/2

barche commented 1 year ago

I filtered out the remaining failing platforms. There are compile errors on Mac, library errors on FreeBSD and for the new M1 macs GDAL isn't built. Up to you if you want to try to fix all that or just disable the platforms, I can help if needed.

sunayana commented 1 year ago

Thanks a lot @barche . I saw the errors too on Mac, I think some were because of stricter compiler checks in Mac. I already saw your changes for disabling those platforms. I am fine with that for the time being. Thanks again.

sunayana commented 1 year ago

It all works at my end so I will go ahead and close this issue. Thank you.