search

JuliaTime / TimeZones.jl

IANA time zone database access for the Julia programming language
Other
86 stars 52 forks source link

Unable to automatically install 'tzdata1997b' from '.../Artifacts.toml' #325

Closed mchitre closed 1 year ago

mchitre commented 3 years ago

Maybe related to #302 ?

But I consistently get this while using PackageCompiler on a pretty much empty project with just 1 dependency (TimeZones.jl):

ERROR: LoadError: Unable to automatically install 'tzdata1997b' from '/home/ubuntu/.julia/packages/TimeZones/K98G0/Artifacts.toml'
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:33
 [2] ensure_artifact_installed(name::String, meta::Dict{String, Any}, artifacts_toml::String; platform::Base.BinaryPlatforms.Platform, verbose::Bool, quiet_download::Bool)
   @ Pkg.Artifacts /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Artifacts.jl:436
 [3] ensure_artifact_installed(name::String, artifacts_toml::String; platform::Base.BinaryPlatforms.Platform, pkg_uuid::Nothing, verbose::Bool, quiet_download::Bool)
   @ Pkg.Artifacts /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Artifacts.jl:400
 [4] ensure_artifact_installed
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Artifacts.jl:395 [inlined]
 [5] bundle_artifacts(ctx::Pkg.Types.Context, app_dir::String)
   @ PackageCompiler ~/.julia/packages/PackageCompiler/3BsME/src/PackageCompiler.jl:770
 [6] create_app(package_dir::String, app_dir::String; app_name::Nothing, precompile_execution_file::Vector{String}, precompile_statements_file::String, incremental::Bool, filter_stdlibs::Bool, audit::Bool, force::Bool, c_driver_program::String, cpu_target::String)
   @ PackageCompiler ~/.julia/packages/PackageCompiler/3BsME/src/PackageCompiler.jl:656
 [7] top-level scope
   @ ~/ttt/test1/src/compile.jl:5
in expression starting at /home/ubuntu/ttt/test1/src/compile.jl:5

Tested on OS X and Linux. Also tested with Julia 1.6 and Julia 1.5.4. Same error on all.

(manually calling build TimeZones works just fine)

PackageCompiler script that fails:

using PackageCompiler
create_app("test1", "test1app"; force=true)
omus commented 3 years ago

Related to: https://github.com/JuliaTime/TimeZones.jl/issues/300

mchitre commented 3 years ago

Indeed using https://github.com/JuliaLang/PackageCompiler.jl/pull/489 fixes the problem

Byrth commented 3 years ago

That allows you to avoid the download, but the failure is due to an unpacking error.

julia > using Pkg
julia> Pkg.PlatformEngines.download_verify_unpack("https://data.iana.org/time-zones/releases/tzdata1997b.tar.gz", "8a6c0801bb2474342a48d6c2253e7983d50d4c6d80a1d455c06fa4e4434df3b3", "/root/.julia/artifacts/jl_rJlNRK"; verbose = true, quiet_download = false)
[ Info: No hash cache found
[ Info: Calculated hash 8a6c0801bb2474342a48d6c2253e7983d50d4c6d80a1d455c06fa4e4434df3b3 for file /tmp/jl_FQy214-download.gz
[ Info: Unpacking /tmp/jl_FQy214-download.gz into /root/.julia/artifacts/jl_rJlNRK...
ERROR: incorrect header checksum = 0; should be 2782
"africa\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0   444 \0 21140 \0     0 \0      53756  6305412777         \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:33
  [2] read_standard_header(io::Base.Process; buf::Vector{UInt8}, tee::Base.DevNull)
    @ Tar /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Tar/src/extract.jl:496
  [3] #read_header#47
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Tar/src/extract.jl:365 [inlined]
  [4] read_tarball(callback::Tar.var"#25#27"{Vector{UInt8}, Bool, Base.Process, String}, predicate::Tar.var"#1#2", tar::Base.Process; buf::Vector{UInt8}, skeleton::Base.DevNull)
    @ Tar /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Tar/src/extract.jl:326
  [5] extract_tarball(predicate::Function, tar::Base.Process, root::String; buf::Vector{UInt8}, skeleton::Base.DevNull, copy_symlinks::Bool)
    @ Tar /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Tar/src/extract.jl:57
  [6] (::Tar.var"#76#79"{String, Base.Process, Tar.var"#1#2"})(skeleton::Base.DevNull)
    @ Tar /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Tar/src/Tar.jl:201
  [7] arg_write(f::Tar.var"#76#79"{String, Base.Process, Tar.var"#1#2"}, arg::Base.DevNull)
    @ ArgTools /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/ArgTools/src/ArgTools.jl:112
  [8] #75
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Tar/src/Tar.jl:200 [inlined]
  [9] arg_mkdir(f::Tar.var"#75#78"{Base.Process, Base.DevNull, Tar.var"#1#2"}, arg::String)
    @ ArgTools /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/ArgTools/src/ArgTools.jl:163
 [10] #74
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Tar/src/Tar.jl:196 [inlined]
 [11] arg_read(f::Tar.var"#74#77"{Base.DevNull, Tar.var"#1#2", String}, arg::Base.Process)
    @ ArgTools /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/ArgTools/src/ArgTools.jl:61
 [12] extract(predicate::Function, tarball::Base.Process, dir::String; skeleton::Nothing, copy_symlinks::Nothing)
    @ Tar /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Tar/src/Tar.jl:195
 [13] #extract#80
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Tar/src/Tar.jl:217 [inlined]
 [14] (::Pkg.PlatformEngines.var"#24#26"{String})(io::Base.Process)
    @ Pkg.PlatformEngines /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/PlatformEngines.jl:512
 [15] open(::Pkg.PlatformEngines.var"#24#26"{String}, ::Cmd; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Base ./process.jl:393
 [16] open
    @ ./process.jl:391 [inlined]
 [17] download_verify_unpack(url::String, hash::String, dest::String; tarball_path::Nothing, ignore_existence::Bool, force::Bool, verbose::Bool, quiet_download::Bool, io::Base.TTY)
    @ Pkg.PlatformEngines /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/PlatformEngines.jl:511
 [18] top-level scope
    @ REPL[19]:1

It looks to me like the read_standard_header() function of the Tar library is throwing an error indicating that this is an invalid Tar. tar -xvzf tzdata1997b.tar.gz has no issue with it and doesn't say anything about a bad header, so I am going to naively report this against the Tar library.

Byrth commented 3 years ago

The summary here is that some of the IANA .tar.gz files from before the year 2000 don't have checksums written to their .tar headers. A workaround is to make a directory in .julia/artifacts, which will block the download/unpack/verify attempt:

# blacklist tzdata 1997b, 95d, 95g 95e, 1999f, 96e, 95i, 93g, 1997c, 96c, 1996n, 94f, 94a,
# 96d, 94d, 95f, 95b, 96b, 96i, 95c, 96a, 96h, 94e, 94h, 95h, 95l, 94b, 95k, 95m
mkdir 00af50ba7c66b49ad97fdf7a529e476aec5039ad \
    4181abd2be1fc8f2964c0681d8bc3ff7d38c299a \
    2ad4eaf2fd0db4fd756019cca85d1139303e0c6f \
    6ac70b261703c39615f2e7797aa3662f57e97a9a \
    02232194ac597e358d30e1f7da36e8ef2ecea1ca \
    d34c0c6aeb16e0a4e1b4b06573459df0f4c86122 \
    d5506bf9dc54bddf87ce34e54075f1bdb8e1e01c \
    e7ddeb4a45080afa6dbdd7c38df81bcf8b9d8f1a \
    191a315cdfccae9f67c584f39c7341182fef2d7d \
    0098bd8f0a47871e728d40fca476b7c42f988894 \
    41df1c2e7f1ab4ae8e9c3ed1904c807238a11e16 \
    120dc3d384840054741fa6d3b1d4bb8eba2396b6 \
    3792c6bcf2dd410edabc0de920b3262e9bcf08b9 \
    c3abe4165d88d36b015eb341b65ff5c5a4c98bf4 \
    ae7b60dc0c96fc76c945b5f95b464c2672a1f12b \
    f052bec9396067db314ae37cba669e1fdb528742 \
    5cee98fd13b7cc03af48b926534759f515d2c1b6 \
    214e8413b5da90d6e5d9f6d916be43b48a9f0669 \
    33143e05ac18f5c2618df3b30ade30b488ecbc83 \
    b973690beafaad07f8fe8172a664e3995665d510 \
    d979d12f0912737480e554e2915459b905220779 \
    463561ed73c004096d46613a9c94af2b635c4ebf \
    90b27ec406fbe1e30a7b486edde037c7fa475c27 \
    a7192b7b16d072863839ee24cc68cb944f942fa4 \ 
    ab0e8405a16af3653a5610979ab2a7392ec4fbba \
    64afaaed2b88c38a29f9fb4198f22662323d3ceb \
    ff665b96f7d50272d0a8e0e321d530c8b09380a5 \
    4641d762bdfd45005ffe23a21bace16fb2e5c50a \
    f7b20d08ea0fae0f43fce023421032eb9c839f27

You can find the expected definition of a .tar header here.

omus commented 1 year ago

In PR https://github.com/JuliaTime/TimeZones.jl/pull/441 all of the lazy tzdata artifacts were removed in favor of using precompiled time zones provided by TZJData.jl non-lazy artifacts. The end result is that when using PackageCompiler it doesn't matter if you use include_lazy=true or not you'll just download the a single tzdata artifact.

Included in TimeZones.jl release 1.12.