search

JuliaGeo / GDAL.jl

Thin Julia wrapper for GDAL - Geospatial Data Abstraction Library
MIT License
90 stars 13 forks source link

GDAL does not compile in Alpine:3.14 Linux docker container #151

Open msbahal opened 1 year ago

msbahal commented 1 year ago

We use a Linux Alpine image for our production servers however, GDAL does not compile on the docker container. The steps to reproduce this are: ` sh> docker pull julia:1.6.2-alpine3.14

sh> docker run -it julia:1.6.2-alpine3.14

julia> ]

Pkg> add GDAL

to see detailed errors run the below

Pkg> precompile `

Test environment:

Things I have tried so far:

Error: `

(@v1.6) pkg> precompile Precompiling project... ✗ PROJ_jll ✗ LittleCMS_jll ✗ OpenJpeg_jll ✗ libgeotiff_jll ✗ GDAL_jll ✗ GDAL 0 dependencies successfully precompiled in 8 seconds (29 already precompiled)

ERROR: The following 1 direct dependency failed to precompile:

GDAL [add2ef01-049f-52c4-9ee2-e494f65e021a]

Failed to precompile GDAL [add2ef01-049f-52c4-9ee2-e494f65e021a] to /root/.julia/compiled/v1.6/GDAL/jl_iHjPOh. ERROR: LoadError: LoadError: InitError: could not load library "/root/.julia/artifacts/4d94fac9395da1d9e54143cd9329aa4ed2882f36/lib/libtiff.so" Error loading shared library libzstd.so.1: No such file or directory (needed by /root/.julia/artifacts/4d94fac9395da1d9e54143cd9329aa4ed2882f36/lib/libtiff.so) Stacktrace: [1] dlopen(s::String, flags::UInt32; throw_error::Bool) @ Base.Libc.Libdl ./libdl.jl:114 [2] dlopen(s::String, flags::UInt32) @ Base.Libc.Libdl ./libdl.jl:114 [3] macro expansion @ ~/.julia/packages/JLLWrappers/QpMQW/src/products/library_generators.jl:54 [inlined] [4] init() @ Libtiff_jll ~/.julia/packages/Libtiff_jll/5VD1s/src/wrappers/x86_64-linux-musl.jl:12 [5] _include_from_serialized(path::String, depmods::Vector{Any}) @ Base ./loading.jl:696 [6] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String) @ Base ./loading.jl:782 [7] _require(pkg::Base.PkgId) @ Base ./loading.jl:1020 [8] require(uuidkey::Base.PkgId) @ Base ./loading.jl:936 [9] require(into::Module, mod::Symbol) @ Base ./loading.jl:923 [10] include(mod::Module, _path::String) @ Base ./Base.jl:386 [11] top-level scope @ ~/.julia/packages/JLLWrappers/QpMQW/src/toplevel_generators.jl:188 [12] include @ ./Base.jl:386 [inlined] [13] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String) @ Base ./loading.jl:1235 [14] top-level scope @ none:1 [15] eval @ ./boot.jl:360 [inlined] [16] eval(x::Expr) @ Base.MainInclude ./client.jl:446 [17] top-level scope @ none:1 during initialization of module Libtiff_jll in expression starting at /root/.julia/packages/PROJ_jll/oT9sV/src/wrappers/x86_64-linux-musl-cxx11.jl:5 in expression starting at /root/.julia/packages/PROJ_jll/oT9sV/src/PROJ_jll.jl:2 ERROR: LoadError: LoadError: Failed to precompile PROJ_jll [58948b4f-47e0-5654-a9ad-f609743f8632] to /root/.julia/compiled/v1.6/PROJ_jll/jl_PjLclO. Stacktrace: [1] error(s::String) @ Base ./error.jl:33 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IOContext{Base.PipeEndpoint}, internal_stdout::IOContext{IOStream}, ignore_loaded_modules::Bool) @ Base ./loading.jl:1385 [3] compilecache(pkg::Base.PkgId, path::String) @ Base ./loading.jl:1329 [4] _require(pkg::Base.PkgId) @ Base ./loading.jl:1043 [5] require(uuidkey::Base.PkgId) @ Base ./loading.jl:936 [6] require(into::Module, mod::Symbol) @ Base ./loading.jl:923 [7] include(mod::Module, _path::String) @ Base ./Base.jl:386 [8] top-level scope @ ~/.julia/packages/JLLWrappers/QpMQW/src/toplevel_generators.jl:188 [9] include @ ./Base.jl:386 [inlined] [10] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String) @ Base ./loading.jl:1235 [11] top-level scope @ none:1 [12] eval @ ./boot.jl:360 [inlined] [13] eval(x::Expr) @ Base.MainInclude ./client.jl:446 [14] top-level scope @ none:1 in expression starting at /root/.julia/packages/GDAL_jll/mKGP0/src/wrappers/x86_64-linux-musl-cxx11.jl:5 in expression starting at /root/.julia/packages/GDAL_jll/mKGP0/src/GDAL_jll.jl:2 ERROR: LoadError: Failed to precompile GDAL_jll [a7073274-a066-55f0-b90d-d619367d196c] to /root/.julia/compiled/v1.6/GDAL_jll/jl_AKFeLI. Stacktrace: [1] error(s::String) @ Base ./error.jl:33 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IOContext{Base.PipeEndpoint}, internal_stdout::IOContext{IOStream}, ignore_loaded_modules::Bool) @ Base ./loading.jl:1385 [3] compilecache(pkg::Base.PkgId, path::String) @ Base ./loading.jl:1329 [4] _require(pkg::Base.PkgId) @ Base ./loading.jl:1043 [5] require(uuidkey::Base.PkgId) @ Base ./loading.jl:936 [6] require(into::Module, mod::Symbol) @ Base ./loading.jl:923 [7] include @ ./Base.jl:386 [inlined] [8] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing) @ Base ./loading.jl:1235 [9] top-level scope @ none:1 [10] eval @ ./boot.jl:360 [inlined] [11] eval(x::Expr) @ Base.MainInclude ./client.jl:446 [12] top-level scope @ none:1 in expression starting at /root/.julia/packages/GDAL/wjbw2/src/GDAL.jl:1

`

evetion commented 1 year ago

So this is on an older Julia version, and probably requires a musl based build. I advise to move to more recent Julia versions, it will probably work out of the box.

visr commented 1 year ago

We do have musl builds for GDAL_jll: https://github.com/JuliaBinaryWrappers/GDAL_jll.jl/releases

It would be good to know what version of GDAL_jll and dependencies you see this on (pkg> st -m), and if you also see it on the latest 1.6 patch release and 1.8.

msbahal commented 1 year ago

@evetion @visr Thanks for the quick responses. I have put together a Dockerfile that I can use to test different combinations of julia and Alpine. All of the docker images fail to build GDAL_jl. I use apk to add musl-dev. I am assuming that is the correct thing to do for the correct GDAL_jll builds (musl) to be pulled?

Dockerfile `

FROM julia:1.6.2-alpine3.14

FROM julia:1.8-alpine3.17

FROM julia:alpine3.17

RUN apk add --update --upgrade --no-cache gcc openssh netcat-openbsd zip unzip curl gnu-libiconv musl-dev postgresql-client aws-cli

RUN adduser -u 1000 -S runtime --home /runtime

ENV JULIA_DEPOT_PATH=/runtime/.julia

WORKDIR /app

RUN julia -e 'using Pkg; Pkg.instantiate()'

RUN julia -e 'using Pkg; Pkg.add("GDAL"); Pkg.precompile()' `

It would be good to know what version of GDAL_jll and dependencies you see this on (pkg> st -m), and if you also see it on the latest 1.6 patch release and 1.8.

Running st -m on julia:1.8-alpine3.17 returns the following. Note - I tried it with and without the apk add step in the Dockerfile above with same result.

(@v1.8) pkg> st -m Status ~/.julia/environments/v1.8/Manifest.toml [fa961155] CEnum v0.4.2 [add2ef01] GDAL v1.5.1 [692b3bcd] JLLWrappers v1.4.1 [21216c6a] Preferences v1.3.0 [8ce61222] Arrow_jll v10.0.0+1 [6e34b625] Bzip2_jll v1.0.8+0 [2e619515] Expat_jll v2.4.8+0 [a7073274] GDAL_jll v301.600.200+0 [d604d12d] GEOS_jll v3.11.2+0 [0234f1f7] HDF5_jll v1.12.2+2 [aacddb02] JpegTurbo_jll v2.1.91+0 [b39eb1a6] Kerberos_krb5_jll v1.19.3+0 [88015f11] LERC_jll v3.0.0+1 [dd4b983a] LZO_jll v2.10.1+0 [08be9ffa] LibPQ_jll v14.3.0+1 [94ce4f54] Libiconv_jll v1.16.1+2 [89763e89] Libtiff_jll v4.4.0+0 [d3a379c0] LittleCMS_jll v2.12.0+0 [5ced341a] Lz4_jll v1.9.3+0 [7243133f] NetCDF_jll v400.902.5+1 [643b3616] OpenJpeg_jll v2.4.0+0 [458c3c95] OpenSSL_jll v1.1.20+0 [58948b4f] PROJ_jll v900.100.0+0 [76ed43ae] SQLite_jll v3.41.0+0 [e0b8ae26] Thrift_jll v0.16.0+0 [02c8fc9c] XML2_jll v2.10.3+0 [3161d3a3] Zstd_jll v1.5.5+0 ⌅ [28df3c45] boost_jll v1.76.0+1 [06c338fa] libgeotiff_jll v100.700.100+0 [b53b4c65] libpng_jll v1.6.38+0 [fe1e1685] snappy_jll v1.1.9+1 [0dad84c5] ArgTools v1.1.1 [56f22d72] Artifacts [2a0f44e3] Base64 [ade2ca70] Dates [f43a241f] Downloads v1.6.0 [7b1f6079] FileWatching [b77e0a4c] InteractiveUtils [b27032c2] LibCURL v0.6.3 [76f85450] LibGit2 [8f399da3] Libdl [56ddb016] Logging [d6f4376e] Markdown [ca575930] NetworkOptions v1.2.0 [44cfe95a] Pkg v1.8.0 [de0858da] Printf [3fa0cd96] REPL [9a3f8284] Random [ea8e919c] SHA v0.7.0 [9e88b42a] Serialization [6462fe0b] Sockets [fa267f1f] TOML v1.0.0 [a4e569a6] Tar v1.10.1 [cf7118a7] UUIDs [4ec0a83e] Unicode [e66e0078] CompilerSupportLibraries_jll v1.0.1+0 [deac9b47] LibCURL_jll v7.84.0+0 [29816b5a] LibSSH2_jll v1.10.2+0 [c8ffd9c3] MbedTLS_jll v2.28.0+0 [14a3606d] MozillaCACerts_jll v2022.2.1 [83775a58] Zlib_jll v1.2.12+3 [8e850ede] nghttp2_jll v1.48.0+0 [3f19e933] p7zip_jll v17.4.0+0

visr commented 1 year ago

Thanks for the info. So this happens on the latest GDAL_jll and latest Julia.

The key error is this:

ERROR: LoadError: LoadError: InitError: could not load library "/root/.julia/artifacts/4d94fac9395da1d9e54143cd9329aa4ed2882f36/lib/libtiff.so"
Error loading shared library libzstd.so.1: No such file or directory (needed by /root/.julia/artifacts/4d94fac9395da1d9e54143cd9329aa4ed2882f36/lib/libtiff.so)

This could be caused by new libzstd builds breaking the libtiff build, you could try Zstd_jll v1.5.2, see if that resolves it. If so, we probably need to trigger a new build of libtiff and possibly GDAL in Yggdrasil.

https://github.com/JuliaBinaryWrappers/Zstd_jll.jl/releases

msbahal commented 1 year ago

@visr How do I enforce the use of Zstd_jll@1.5.2. I tried the following inside a Julia project within an Alpine docker container and it didn't work:

bash> docker run -it julia:1.8-alpine3.17 julia> generate testProject shell> cd testProject pkg> activate . ipkg> instantiate pkg> add Zstd_jll@1.5.2 pkg> compat ## then pin the Zstd_jll to 1.5.2 pkg> add GDAL

Precompilation fails with same libzstd.so error. I tries using an older version of Libtiff_jll as well using the same method above but that didn't work either.

Output of st -m is below [fa961155] CEnum v0.4.2 [add2ef01] GDAL v1.5.1 [692b3bcd] JLLWrappers v1.4.1 [21216c6a] Preferences v1.3.0 [8ce61222] Arrow_jll v10.0.0+1 [6e34b625] Bzip2_jll v1.0.8+0 [2e619515] Expat_jll v2.4.8+0 [a7073274] GDAL_jll v301.600.200+0 [d604d12d] GEOS_jll v3.11.2+0 [0234f1f7] HDF5_jll v1.12.2+2 [aacddb02] JpegTurbo_jll v2.1.91+0 [b39eb1a6] Kerberos_krb5_jll v1.19.3+0 [88015f11] LERC_jll v3.0.0+1 [dd4b983a] LZO_jll v2.10.1+0 [08be9ffa] LibPQ_jll v14.3.0+1 [94ce4f54] Libiconv_jll v1.16.1+2 [89763e89] Libtiff_jll v4.4.0+0 [d3a379c0] LittleCMS_jll v2.12.0+0 [5ced341a] Lz4_jll v1.9.3+0 [7243133f] NetCDF_jll v400.902.5+1 [643b3616] OpenJpeg_jll v2.4.0+0 [458c3c95] OpenSSL_jll v1.1.20+0 [58948b4f] PROJ_jll v900.100.0+0 [76ed43ae] SQLite_jll v3.41.0+0 [e0b8ae26] Thrift_jll v0.16.0+0 [02c8fc9c] XML2_jll v2.10.3+0 ⌅ [3161d3a3] Zstd_jll v1.5.2+0 ⌅ [28df3c45] boost_jll v1.76.0+1 [06c338fa] libgeotiff_jll v100.700.100+0 [b53b4c65] libpng_jll v1.6.38+0 [fe1e1685] snappy_jll v1.1.9+1 [0dad84c5] ArgTools v1.1.1 [56f22d72] Artifacts [2a0f44e3] Base64 [ade2ca70] Dates [f43a241f] Downloads v1.6.0 [7b1f6079] FileWatching [b77e0a4c] InteractiveUtils [b27032c2] LibCURL v0.6.3 [76f85450] LibGit2 [8f399da3] Libdl [56ddb016] Logging [d6f4376e] Markdown [ca575930] NetworkOptions v1.2.0 [44cfe95a] Pkg v1.8.0 [de0858da] Printf [3fa0cd96] REPL [9a3f8284] Random [ea8e919c] SHA v0.7.0 [9e88b42a] Serialization [6462fe0b] Sockets [fa267f1f] TOML v1.0.0 [a4e569a6] Tar v1.10.1 [cf7118a7] UUIDs [4ec0a83e] Unicode [e66e0078] CompilerSupportLibraries_jll v1.0.1+0 [deac9b47] LibCURL_jll v7.84.0+0 [29816b5a] LibSSH2_jll v1.10.2+0 [c8ffd9c3] MbedTLS_jll v2.28.0+0 [14a3606d] MozillaCACerts_jll v2022.2.1 [83775a58] Zlib_jll v1.2.12+3 [8e850ede] nghttp2_jll v1.48.0+0 [3f19e933] p7zip_jll v17.4.0+0

Below is what the Project.toml looks like

name = "HW" uuid = "08117d24-684c-431d-be48-384bc248f60c" authors = ["Unknown "] version = "0.1.0"

[deps] GDAL = "add2ef01-049f-52c4-9ee2-e494f65e021a" Zstd_jll = "3161d3a3-bdf6-5164-811a-617609db77b4"

[compat] Zstd_jll = "=1.5.2"

visr commented 1 year ago

How do I enforce the use of Zstd_jll@1.5.2.

What you did worked to get that version, although unfortunately it doesn't seem to work.

I saw you opened the same issue in Yggdrasil. Probably the workaround suggested there would be the next thing to try: https://github.com/JuliaPackaging/Yggdrasil/issues/6557#issuecomment-1506010315.

msbahal commented 1 year ago

Thanks @visr . I'll give that workaround a go and see if that fixes things. I'm not sure if I will be able to use that workaround in our production system but atleast that will help us pinpoint the issue. I'll update here once I've tried that.