search

JuliaPackaging / BinaryBuilder.jl

Binary Dependency Builder for Julia
https://binarybuilder.org
Other
392 stars 101 forks source link

Inconsistent version of dependency #1210

Closed Alexander-Barth closed 2 years ago

Alexander-Barth commented 2 years ago

I am trying to compile a package (NetCDF_jll) with the a dependency on LibCURL for Julia 1.8:

 jll_stdlibs = Dict(
        v"1.3" => [
            Dependency("LibCURL_jll"; compat = "~7.71.1"),
        ],
        v"1.6" => [
            Dependency("LibCURL_jll"; compat = "~7.73.0"),
        ],
        v"1.8" => [
            Dependency("LibCURL_jll"; compat = "~7.81.0"), # also tried Dependency("LibCURL_jll",  v"7.81.0")
        ]
    )
   # min_julia_version = v"1.8"
    append!(dependencies, jll_stdlibs[min_julia_version])

I would expect that all compile targets would be build using LibCURL 7.81.0 for julia 1.8. This is for instant the case when I just compile for x86_64-linux-gnu:

julia --color=yes ./build_tarballs.jl x86_64-linux-gnu --debug --verbose | tee x86_64-linux-gnu.log
$ grep LibCURL_jll x86_64-linux-gnu.log
  [deac9b47] + LibCURL_jll v7.81.0+0
  [deac9b47] + LibCURL_jll v7.81.0+0

But if I choose 2 (or more) build targets at the same time, I get inconsistent results:

julia --color=yes ./build_tarballs.jl x86_64-linux-gnu,x86_64-apple-darwin --debug --verbose | tee x86_64-linux-gnu+x86_64-apple-darwin3.log
$ grep LibCURL_jll x86_64-linux-gnu+x86_64-apple-darwin3.log
  [deac9b47] + LibCURL_jll v7.81.0+0  # <- x86_64-apple-darwin
  [deac9b47] + LibCURL_jll v7.81.0+0
  [deac9b47] + LibCURL_jll v7.83.1+1 # <- x86_64-linux-gnu
  [deac9b47] + LibCURL_jll v7.83.1+1

x86_64-linux-gnu was now compiled with LibCURL v7.83.1+1 instead of v7.81.0+0.

More verbose output of the 2nd test (for x86_64-linux-gnu) :

  Resolving package versions...
    Updating `/mnt/data1/abarth/src/Yggdrasil/N/NetCDF/NetCDF@julia-1.8/build/x86_64-linux-gnu/pgNMu7Xf/x86_64-linux-gnu-libgfortran3-cxx03/.project/Project.toml`
  [0234f1f7] + HDF5_jll v1.12.1+0
  [02c8fc9c] + XML2_jll v2.9.14+0
  [deac9b47] + LibCURL_jll v7.83.1+1
  [83775a58] + Zlib_jll v1.2.12+3
    Updating `/mnt/data1/abarth/src/Yggdrasil/N/NetCDF/NetCDF@julia-1.8/build/x86_64-linux-gnu/pgNMu7Xf/x86_64-linux-gnu-libgfortran3-cxx03/.project/Manifest.toml`
[...]
  [b27032c2] + LibCURL v0.6.3
[...]
  [deac9b47] + LibCURL_jll v7.83.1+1

I am using BinaryBuilder v0.5.5, BinaryBuilderBase v1.12.0 and Julia v1.7.2 on Linux.

giordano commented 2 years ago

Grepping without context isn't very useful. There are multiple environments, from the code you shared I have no idea what is what.

Alexander-Barth commented 2 years ago

Here are the full log files for context: x86_64-linux-gnu.log x86_64-linux-gnu+x86_64-apple-darwin3.log

The problematic environment is this one (x86_64-linux-gnu called after x86_64-apple-darwin) in the file x86_64-linux-gnu+x86_64-apple-darwin3.log :

    Updating `/mnt/data1/abarth/src/Yggdrasil/N/NetCDF/NetCDF@julia-1.8/build/x86_64-linux-gnu/pgNMu7Xf/x86_64-linux-gnu-libgfortran3-cxx03/.project/Project.toml`
  [0234f1f7] + HDF5_jll v1.12.1+0
  [02c8fc9c] + XML2_jll v2.9.14+0
  [deac9b47] + LibCURL_jll v7.83.1+1 # <- expected to see 7.81.0
  [83775a58] + Zlib_jll v1.2.12+3
    Updating `/mnt/data1/abarth/src/Yggdrasil/N/NetCDF/NetCDF@julia-1.8/build/x86_64-linux-gnu/pgNMu7Xf/x86_64-linux-gnu-libgfortran3-cxx03/.project/Manifest.toml`
  [34da2185] + Compat v2.2.1
  [692b3bcd] + JLLWrappers v1.4.1
  [21216c6a] + Preferences v1.3.0
  [0234f1f7] + HDF5_jll v1.12.1+0
  [94ce4f54] + Libiconv_jll v1.16.1+1
  [458c3c95] + OpenSSL_jll v1.1.14+0
  [02c8fc9c] + XML2_jll v2.9.14+0
  [0dad84c5] + ArgTools v1.1.1
  [56f22d72] + Artifacts v1.3.0
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [8bb1440f] + DelimitedFiles
  [8ba89e20] + Distributed
  [f43a241f] + Downloads v1.6.0
  [7b1f6079] + FileWatching
  [b77e0a4c] + InteractiveUtils
  [b27032c2] + LibCURL v0.6.3
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [a63ad114] + Mmap
  [ca575930] + NetworkOptions v1.2.0
  [44cfe95a] + Pkg
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA v0.5.7
  [9e88b42a] + Serialization
  [1a1011a3] + SharedArrays
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics
  [fa267f1f] + TOML v1.0.3
  [a4e569a6] + Tar v1.9.2
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [deac9b47] + LibCURL_jll v7.83.1+1
  [29816b5a] + LibSSH2_jll v1.10.2+0
  [c8ffd9c3] + MbedTLS_jll v2.28.0+0
  [14a3606d] + MozillaCACerts_jll v2022.3.29+0
  [83775a58] + Zlib_jll v1.2.12+3
  [8e850b90] + libblastrampoline_jll v5.1.0+0
  [8e850ede] + nghttp2_jll v1.47.0+0
  [3f19e933] + p7zip_jll v17.4.0+0

If needed, the full build scripts are here:

https://github.com/Alexander-Barth/Yggdrasil/blob/35637c85e665950d606ced091a9cb5621c67a34e/N/NetCDF/common.jl https://github.com/Alexander-Barth/Yggdrasil/blob/35637c85e665950d606ced091a9cb5621c67a34e/N/NetCDF/NetCDF%40julia-1.8/build_tarballs.jl

giordano commented 2 years ago

I haven't looked at the problem yet, but I would say this isn't a blocker for Yggdrasil, since we always build a single target platform

giordano commented 2 years ago

Much smaller reproducer instead of a full blown multi-file recipe:

using BinaryBuilder

name = "libfoo"
version = v"1.0.0"

sources = ArchiveSource[]

script = ""

platforms = supported_platforms()

products = LibraryProduct[]

dependencies = [
    Dependency("LibCURL_jll"; compat = "7.81.0"),
]

build_tarballs(ARGS, name, version, sources, script, platforms, products, dependencies;
               julia_compat="1.8", skip_audit=true)

Build with

julia build_tarballs.jl --debug --verbose x86_64-linux-gnu,x86_64-apple-darwin

as suggested above and you can see the second build gets the wrong version.

But I believe this is a bug in Pkg:

julia> using Pkg

julia> ctx = Pkg.Types.Context(; julia_version=nothing);

# `cab60d2381bf2a404bcccccb7b3f861aafcb3ad8` is the tree-hash of `7.81.0+0`: https://github.com/JuliaRegistries/General/blob/18fa4e80204518bf81e44b097d90559f6bb08353/L/LibCURL_jll/Versions.toml#L43-L44
julia> Pkg.add(ctx, [Pkg.Types.PackageSpec(; name = "LibCURL_jll", tree_hash = Base.SHA1("cab60d2381bf2a404bcccccb7b3f861aafcb3ad8"))])
    Updating registry at `~/.julia/registries/General`
    Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Resolving package versions...
    Updating `/tmp/jl_AjKnQ7/Project.toml`
  [deac9b47] + LibCURL_jll v7.83.1+1
    Updating `/tmp/jl_AjKnQ7/Manifest.toml`
  [34da2185] + Compat v2.2.1
  [692b3bcd] + JLLWrappers v1.4.1
  [21216c6a] + Preferences v1.3.0
  [0dad84c5] + ArgTools v1.1.1
  [56f22d72] + Artifacts v1.3.0
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [8bb1440f] + DelimitedFiles
  [8ba89e20] + Distributed
  [f43a241f] + Downloads v1.6.0
  [7b1f6079] + FileWatching
  [b77e0a4c] + InteractiveUtils
  [b27032c2] + LibCURL v0.6.3
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [a63ad114] + Mmap
  [ca575930] + NetworkOptions v1.2.0
  [44cfe95a] + Pkg
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA v0.5.7
  [9e88b42a] + Serialization
  [1a1011a3] + SharedArrays
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics
  [fa267f1f] + TOML v1.0.3
  [a4e569a6] + Tar v1.9.2
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [deac9b47] + LibCURL_jll v7.83.1+1
  [29816b5a] + LibSSH2_jll v1.10.2+0
  [c8ffd9c3] + MbedTLS_jll v2.28.0+0
  [14a3606d] + MozillaCACerts_jll v2022.3.29+0
  [83775a58] + Zlib_jll v1.2.12+3
  [8e850b90] + libblastrampoline_jll v5.1.0+0
  [8e850ede] + nghttp2_jll v1.47.0+0
  [3f19e933] + p7zip_jll v17.4.0+0

which I'm not very inclined to work around.

giordano commented 2 years ago

To add more fun to whoever is going to investigate this issue further, that is not me:

julia> using BinaryBuilderBase, Pkg

julia> Pkg.activate(; temp=true)
  Activating new project at `/tmp/jl_91kNvw`

julia> ctx = Pkg.Types.Context(; julia_version=nothing);

julia> spec = BinaryBuilderBase.get_addable_spec("LibCURL_jll", v"7.81.0+0")
PackageSpec(
  name = LibCURL_jll
  uuid = deac9b47-8bc7-5906-a0fe-35ac56dc84c0
  tree_hash = cab60d2381bf2a404bcccccb7b3f861aafcb3ad8
  version = *
  repo/source = "https://github.com/JuliaBinaryWrappers/LibCURL_jll.jl.git"
  repo/rev = 4b758d9a039000f3344c8cbb38031b06f1e5148b
)

julia> Pkg.add(ctx, [spec])
    Updating registry at `~/.julia/registries/General`
    Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Resolving package versions...
    Updating `/tmp/jl_91kNvw/Project.toml`
  [deac9b47] + LibCURL_jll v7.73.0+4 `https://github.com/JuliaBinaryWrappers/LibCURL_jll.jl.git#4b758d9`

I have no idea what to do to make Pkg install what we ask.

Edit issue opened at https://github.com/JuliaLang/Pkg.jl/issues/3113.

giordano commented 2 years ago

The only issue I can see on the BB side is that the first time the vector of dependencies which is passed to BinaryBuilderBase.setup_dependencies is

dependencies = Pkg.Types.PackageSpec[PackageSpec(
  name = LibCURL_jll
  uuid = deac9b47-8bc7-5906-a0fe-35ac56dc84c0
  version = VersionSpec("7.81.0")
)]

and the second time is

dependencies = Pkg.Types.PackageSpec[PackageSpec(
  name = LibCURL_jll
  uuid = deac9b47-8bc7-5906-a0fe-35ac56dc84c0
  tree_hash = cab60d2381bf2a404bcccccb7b3f861aafcb3ad8
  version = v"7.81.0+0"
)]

I have no clue why this is different, I don't expect this vector to change at any point, so that's something that might be fixed, although the second time the vector is still "correct", it isn't like we have to fix the content of the vector but just keep the same one. Again, not a blocker for Yggdrasil.

But it's still rather frustrating that Pkg installs rather arbitrary versions when we ask for something very specific, and I personally have no intention of debugging that.

Alexander-Barth commented 2 years ago

If it is helpful, if I use a deep copy of the dependencies (at this line), then libcurl 7.81.0 gets installed in all builds:

Pkg_add(ctx, deepcopy(dependencies); platform=platform, io=outs)
giordano commented 2 years ago

That sounds like yet another Pkg bug which modifies the input. That's precisely the reason why dependencies is changing, which I mentioned above.

giordano commented 2 years ago

BTW, reproducer:

julia> using Pkg, UUIDs

julia> Pkg.activate(; temp=true)
  Activating new project at `/tmp/jl_d7lyEq`

julia> spec = Pkg.PackageSpec(; name = "LibCURL_jll", uuid = UUID("deac9b47-8bc7-5906-a0fe-35ac56dc84c0"), version = Pkg.Types.VersionSpec("7.81.0"));

julia> ctx = Pkg.Types.Context(; julia_version=nothing);

julia> Pkg.add(ctx, [spec]; io=devnull)

julia> spec
PackageSpec(
  name = LibCURL_jll
  uuid = deac9b47-8bc7-5906-a0fe-35ac56dc84c0
  tree_hash = cab60d2381bf2a404bcccccb7b3f861aafcb3ad8
  version = v"7.81.0+0"
)

spec is modified in-place without notice. It acquired the tree hash, which then causes the problem above of installing the wrong version when the tree hash is used.

Edit: Issue opened at https://github.com/JuliaLang/Pkg.jl/issues/3112