Closed mkitti closed 1 year ago
Here's a repo to test: https://github.com/mkitti/HDF5_jll.jl
No mac support yet.
Can I somehow help with the Mac binary?
Grab this file and test it:
Try passing julia build_tarballs.jl --help
to start.
I bump into an Error: LoadError: unknown archive format
░ tamasgal@silentbox:tmp/hdf5 main ● ஃ v1.8.5 took 8s
░ 16:03:49 > julia build_tarballs.jl aarch64-apple-darwin
┌ Warning: Running Registrator on a Julia version that does not support weak dependencies. Weak dependencies will not be registered.
└ @ RegistryTools ~/.julia/packages/RegistryTools/tHkoa/src/RegistryTools.jl:18
[ Info: Building for aarch64-apple-darwin
[ Info: Downloading https://mirror.msys2.org/mingw/mingw32/mingw-w64-i686-hdf5-1.14.0-3-any.pkg.tar.zst to /Users/tamasgal/.julia/packages/BinaryBuilderBase/Toc7t/deps/downloads/b7fd69d444a9d1f71335c53ab6bfd1449eed9023133758130a042ed3f2c4504a-mingw-w64-i686-hdf5-1.14.0-3-any.pkg.tar.zst...
[ Info: Downloading https://mirror.msys2.org/mingw/mingw32/mingw-w64-i686-zlib-1.2.13-3-any.pkg.tar.zst to /Users/tamasgal/.julia/packages/BinaryBuilderBase/Toc7t/deps/downloads/ed62c6f77f9cce488aed15726349d5d4537689583caab46bace8d41173db48b7-mingw-w64-i686-zlib-1.2.13-3-any.pkg.tar.zst...
[ Info: Downloading https://mirror.msys2.org/mingw/mingw32/mingw-w64-i686-openssl-3.0.8-1-any.pkg.tar.zst to /Users/tamasgal/.julia/packages/BinaryBuilderBase/Toc7t/deps/downloads/882b570d7a53cd2b12a4cbf44712e2662fe0c9f6e1a4cf6872c2467ebda36fbb-mingw-w64-i686-openssl-3.0.8-1-any.pkg.tar.zst...
[ Info: Downloading https://mirror.msys2.org/mingw/mingw32/mingw-w64-i686-gcc-libs-12.2.0-10-any.pkg.tar.zst to /Users/tamasgal/.julia/packages/BinaryBuilderBase/Toc7t/deps/downloads/f181e46ff9dce67379c51aa5f39b1c8dd42617f184d13639f53d9768379bc2b8-mingw-w64-i686-gcc-libs-12.2.0-10-any.pkg.tar.zst...
[ Info: Downloading https://mirror.msys2.org/mingw/mingw64/mingw-w64-x86_64-hdf5-1.14.0-3-any.pkg.tar.zst to /Users/tamasgal/.julia/packages/BinaryBuilderBase/Toc7t/deps/downloads/c4a889e09b43e0ec1f9a872acbfc62f34291ee935c26ea90ca9911104f9627d3-mingw-w64-x86_64-hdf5-1.14.0-3-any.pkg.tar.zst...
[ Info: Downloading https://mirror.msys2.org/mingw/mingw64/mingw-w64-x86_64-zlib-1.2.13-3-any.pkg.tar.zst to /Users/tamasgal/.julia/packages/BinaryBuilderBase/Toc7t/deps/downloads/7fc6ac1629180e205f0fdbe7abd04353136a44d73d16924f0c64fd10828329a7-mingw-w64-x86_64-zlib-1.2.13-3-any.pkg.tar.zst...
[ Info: Downloading https://mirror.msys2.org/mingw/mingw64/mingw-w64-x86_64-openssl-3.0.8-1-any.pkg.tar.zst to /Users/tamasgal/.julia/packages/BinaryBuilderBase/Toc7t/deps/downloads/36040fa35e6990fbfb19a3dd083fe13656da1d1b7d5b5e22c3da96005a1d6687-mingw-w64-x86_64-openssl-3.0.8-1-any.pkg.tar.zst...
[ Info: Downloading https://anaconda.org/conda-forge/hdf5/1.14.0/download/linux-64/hdf5-1.14.0-nompi_h5231ba7_102.conda to /Users/tamasgal/.julia/packages/BinaryBuilderBase/Toc7t/deps/downloads/d64e2e691205920a0d0f15876d4bcade18f98ef126959d21316a297516476c7c-hdf5-1.14.0-nompi_h5231ba7_102.conda...
[ Info: Downloading https://anaconda.org/conda-forge/hdf5/1.14.0/download/linux-aarch64/hdf5-1.14.0-nompi_h4e7b029_102.conda to /Users/tamasgal/.julia/packages/BinaryBuilderBase/Toc7t/deps/downloads/127b4b6a0323c504b197f1ee8e05f9857a57bd51331de15937c8a9cb7b3a302c-hdf5-1.14.0-nompi_h4e7b029_102.conda...
[ Info: Downloading https://anaconda.org/conda-forge/hdf5/1.14.0/download/osx-64/hdf5-1.14.0-nompi_h058c35b_102.conda to /Users/tamasgal/.julia/packages/BinaryBuilderBase/Toc7t/deps/downloads/60a1cf488de41fd7353a0775b749031c91a060ac6c957c93349f482179ed9a89-hdf5-1.14.0-nompi_h058c35b_102.conda...
[ Info: Downloading https://anaconda.org/conda-forge/hdf5/1.14.0/download/osx-arm64/hdf5-1.14.0-nompi_hc28a057_102.conda to /Users/tamasgal/.julia/packages/BinaryBuilderBase/Toc7t/deps/downloads/1cb95014e706b832459784975947a958171ea9903aae077565f775d6f3154fad-hdf5-1.14.0-nompi_hc28a057_102.conda...
ERROR: LoadError: Unknown archive format
Stacktrace:
[1] error(s::String)
@ Base ./error.jl:35
[2] (::BinaryBuilderBase.var"#80#82"{String, BinaryBuilderBase.SetupSource{ArchiveSource}, Bool})()
@ BinaryBuilderBase ~/.julia/packages/BinaryBuilderBase/Toc7t/src/Prefix.jl:339
[3] cd(f::BinaryBuilderBase.var"#80#82"{String, BinaryBuilderBase.SetupSource{ArchiveSource}, Bool}, dir::String)
@ Base.Filesystem ./file.jl:112
[4] #setup#79
@ ~/.julia/packages/BinaryBuilderBase/Toc7t/src/Prefix.jl:300 [inlined]
[5] setup
@ ~/.julia/packages/BinaryBuilderBase/Toc7t/src/Prefix.jl:295 [inlined]
[6] setup_workspace(build_path::String, sources::Vector{BinaryBuilderBase.SetupSource{ArchiveSource}}, target_platform::Platform, host_platform::Platform; verbose::Bool)
@ BinaryBuilderBase ~/.julia/packages/BinaryBuilderBase/Toc7t/src/Prefix.jl:418
[7] autobuild(dir::AbstractString, src_name::AbstractString, src_version::VersionNumber, sources::Vector{<:BinaryBuilderBase.AbstractSource}, script::AbstractString, platforms::Vector, products::Vector{<:Product}, dependencies::Vector{<:BinaryBuilderBase.AbstractDependency}; verbose::Bool, debug::Bool, skip_audit::Bool, ignore_audit_errors::Bool, autofix::Bool, code_dir::Union{Nothing, String}, require_license::Bool, kwargs::Base.Pairs{Symbol, V, Tuple{Vararg{Symbol, N}}, NamedTuple{names, T}} where {V, N, names, T<:Tuple{Vararg{Any, N}}})
@ BinaryBuilder ~/.julia/packages/BinaryBuilder/0CUml/src/AutoBuild.jl:783
[8] build_tarballs(ARGS::Any, src_name::Any, src_version::Any, sources::Any, script::Any, platforms::Any, products::Any, dependencies::Any; julia_compat::String, kwargs::Base.Pairs{Symbol, V, Tuple{Vararg{Symbol, N}}, NamedTuple{names, T}} where {V, N, names, T<:Tuple{Vararg{Any, N}}})
@ BinaryBuilder ~/.julia/packages/BinaryBuilder/0CUml/src/AutoBuild.jl:347
[9] top-level scope
@ ~/tmp/hdf5/build_tarballs.jl:118
in expression starting at /Users/tamasgal/tmp/hdf5/build_tarballs.jl:118
Dev BinaryBuilderBase in your environment.
Ah my bad ;) thanks
OK, got a bit further but the GCCBootstrap is making trouble now:
Downloading artifact: PlatformSupport-aarch64-apple-darwin20.v2021.8.10.x86_64-linux-musl.unpacked
Downloaded artifact: PlatformSupport-aarch64-apple-darwin20.v2021.8.10.x86_64-linux-musl.unpacked
Downloading artifact: GCCBootstrap-aarch64-apple-darwin20.v12.0.1-iains.x86_64-linux-musl.unpacked
Downloaded artifact: GCCBootstrap-aarch64-apple-darwin20.v12.0.1-iains.x86_64-linux-musl.unpacked
Downloading artifact: GCCBootstrap-aarch64-apple-darwin20.v12.0.1-iains.x86_64-linux-musl.unpacked
[ Info: No hash cache found
[ Info: Calculated hash d6ab9dc831485214eac9995e8614d34cfd2cc9f806e00b0cfdb8c425cbbc4520 for file /var/folders/84/mcvklq757tq1nfrkbxvvbq8m0000gn/T/jl_mnUD65yluB-download.gz
Downloaded artifact: GCCBootstrap-aarch64-apple-darwin20.v12.0.1-iains.x86_64-linux-musl.unpacked
ERROR: LoadError: Unable to automatically download/install artifact 'GCCBootstrap-aarch64-apple-darwin20.v12.0.1-iains.x86_64-linux-musl.unpacked' from sources listed in '/Users/tamasgal/.julia/dev/BinaryBuilderBase/Artifacts.toml'.
Sources attempted:
- https://pkg.julialang.org/artifact/d6f2dc0e73e8796cb9bb992a4970412bc9e4cea3
Error: AssertionError: size == padded_size == 0
- https://github.com/JuliaPackaging/Yggdrasil/releases/download/GCCBootstrap-v12.0.1-iains/GCCBootstrap-aarch64-apple-darwin20.v12.0.1-iains.x86_64-linux-musl.unpacked.tar.gz
Error: AssertionError: size == padded_size == 0
You might want to restrict the platforms you are building for. See julia build_tarballs.jl --help
for options.
Hmm... you know what, I think we need more help from @giordano . Maybe try #binarybuilder on Slack.
Yeah, I already restricted by calling julia build_tarballs.jl aarch64-apple-darwin
. OK I'll try on Slack later!
Strange, I can download https://github.com/JuliaPackaging/Yggdrasil/releases/download/GCCBootstrap-v12.0.1-iains/GCCBootstrap-aarch64-apple-darwin20.v12.0.1-iains.x86_64-linux-musl.unpacked.tar.gz
Maybe it's some transient issue?
@tamasgal Valentin had a similar problem a few days ago: he had the disk full, and those large artifacts got truncated. Also, if you are having troubles setting up BinaryBuilder locally, GitHub CodeSpaces might be a much easier option.
Ah thanks, totally my fault. Did not realise that it requires a few GB and did not even notice that the disk was going to fill up.
OK, managed to do it. Sorry for my ignorance but how can I run the HDF5.jl test suite using these binaries?
HDF5-logs.v1.14.0.aarch64-apple-darwin.tar.gz HDF5.v1.14.0.aarch64-apple-darwin.tar.gz
Run build_tarballs.jl with --deploy=local
.
--deploy=<repo> Deploy binaries and JLL wrapper code to a github
release of an autogenerated repository. Uses
github.com/JuliaBinaryWrappers/<name>_jll.jl
by default, unless<repo>
is set, in which case it should be set as<owner>/<name>_jll.jl
. Setting this option is equivalent to setting--deploy-bin
and--deploy-jll
. If<repo>
is set to "local" then nothing will be uploaded, but JLL packages will still be written out to~/.julia/dev/
.
See
$ julia --project=. build_tarballs.jl --help
Usage: build_tarballs.jl [target1,target2,...] [--help]
[--verbose] [--debug]
[--deploy] [--deploy-bin] [--deploy-jll]
[--register] [--meta-json]
...
To spell it out, if you run julia build_tarballs.jl --deploy=local
there should be a ~/.julia/dev/HDF5_jll
.
Then create an environment to test the combination.
using Pkg
pkg"activate --temp"
pkg"dev HDF5_jll"
pkg"add HDF5"
pkg"test HDF5"
Yep, that's what I am doing right now :)
Btw. is the HDF5_jll 1.14.0
only supposed to work with HDF5.jl@1.14
? At least Pkg
complains that there are compatibility issues:
(jl_MBgwbM) pkg> add HDF5@0.16.14
Resolving package versions...
ERROR: Unsatisfiable requirements detected for package HDF5 [f67ccb44]:
HDF5 [f67ccb44] log:
├─possible versions are: 0.9.4-0.16.14 or uninstalled
├─restricted to versions 0.16.14 by an explicit requirement, leaving only versions 0.16.14
└─restricted by compatibility requirements with HDF5_jll [0234f1f7] to versions: [0.9.4-0.13.6, 0.14.0-0.14.1] or uninstalled — no versions left
└─HDF5_jll [0234f1f7] log:
├─possible versions are: 1.14.0 or uninstalled
└─HDF5_jll [0234f1f7] is fixed to version 1.14.0+0
anyways, the test with HDF5.jl@1.14.1
are fine:
Testing Running tests...
HDF5 version 1.14.0
Test Summary: | Pass Broken Total Time
HDF5.jl | 519 1 520 16.8s
Testing HDF5 tests passed
(jl_MBgwbM) pkg> st
Status `/private/var/folders/84/mcvklq757tq1nfrkbxvvbq8m0000gn/T/jl_MBgwbM/Project.toml`
⌃ [f67ccb44] HDF5 v0.14.1
[0234f1f7] HDF5_jll v1.14.0+0 `~/.julia/dev/HDF5_jll`
Info Packages marked with ⌃ have new versions available and may be upgradable.
Oh right, see my mkitti/hdf5_1_14_compat
branch.
https://github.com/mkitti/HDF5.jl/tree/mkitti/hdf5_1_14_compat
Almost there! There is another version check which fails on 1.14
[ Info: libhdf5 v1.14.0
properties: Test Failed at /Users/tamasgal/.julia/packages/HDF5/pbt8t/test/properties.jl:73
Expression: fapl.libver_bounds == (:earliest, Base.thisminor(HDF5.libversion))
Evaluated: (:earliest, v"1.12.0") == (:earliest, v"1.14.0")
Stacktrace:
[1] macro expansion
@ ~/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/share/julia/stdlib/v1.8/Test/src/Test.jl:464 [inlined]
[2] (::var"#323#324")(hfile::HDF5.File)
@ Main ~/.julia/packages/HDF5/pbt8t/test/properties.jl:73
properties: Error During Test at /Users/tamasgal/.julia/packages/HDF5/pbt8t/test/properties.jl:99
Unexpected Pass
Expression: dapl.virtual_view == :last_available
Got correct result, please change to @test if no longer broken.
┌ Warning: `filter=(307, 0)` keyword option is deprecated, use `filters=(307, 0)` instead
│ caller = setproperty!(p::HDF5.DatasetCreateProperties, name::Symbol, val::Tuple{Int32, Int64}) at properties.jl:59
└ @ HDF5 ~/.julia/packages/HDF5/pbt8t/src/properties.jl:59
┌ Warning: `append!(filters::Filters.FilterPipeline, extra::NTuple{N, Integer}) where N` is deprecated, use `append!(filters, [ExternalFilter(extra...)])` instead.
│ caller = set_filters! at properties.jl:497 [inlined]
└ @ Core ~/.julia/packages/HDF5/pbt8t/src/properties.jl:497
Test Summary: | Pass Fail Error Broken Total Time
HDF5.jl | 1105 1 1 3 1110 40.8s
plain | 151 1 152 7.7s
complex | 13 13 0.3s
undefined and null | 4 4 0.0s
abstract arrays | 2 2 0.2s
empty and 0-size arrays | 39 39 0.2s
generic read of native types | 17 17 0.2s
show | 51 51 0.9s
split1 | 13 13 0.1s
haskey | 18 18 0.1s
AbstractString | 51 51 1.3s
opaque data | 7 7 0.3s
FixedStrings and FixedArrays | 18 18 0.3s
Object Exists | 6 6 0.0s
HDF5 existance | 4 4 0.0s
bounds | 2 2 0.1s
h5a_iterate | 8 8 0.1s
h5l_iterate | 8 8 0.1s
compound | 10 10 1.1s
write_compound | 18 18 0.6s
custom | 6 6 0.2s
reference | 6 6 0.1s
Dataspaces | 91 91 0.1s
Datatypes | 15 15 0.0s
BlockRange | 35 35 0.1s
hyperslab | 5 5 0.4s
read 0-length arrays: issue #859 | None 0.1s
attrs interface | 92 92 0.4s
readremote | 23 23 0.5s
extend | 29 29 0.5s
gc | 101 101 1.1s
external | 6 6 0.0s
swmr | 4 4 2.6s
mmap | 9 9 0.3s
properties | 41 1 1 1 44 0.9s
filter | 49 49 6.0s
Raw Chunk I/O | 52 52 1.4s
fileio | 6 6 0.6s
track order | 18 18 0.7s
h5f_get_dset_no_attrs_hint | 6 6 0.1s
non-allocating methods | 11 1 12 0.4s
Compression Filter Unit Tests | 6 6 0.0s
Object API | 38 38 0.2s
virtual dataset | 5 5 0.3s
mpio | 1 1 0.2s
ros3 | 2 2 2.7s
ERROR: LoadError: Some tests did not pass: 1105 passed, 1 failed, 1 errored, 3 broken.
in expression starting at /Users/tamasgal/.julia/packages/HDF5/pbt8t/test/runtests.jl:18
ERROR: Package HDF5 errored during testing
(jl_MBgwbM) pkg> st
Status `/private/var/folders/84/mcvklq757tq1nfrkbxvvbq8m0000gn/T/jl_MBgwbM/Project.toml`
[f67ccb44] HDF5 v0.16.14 `../../../../../../../Users/tamasgal/tmp/hdf5/HDF5.jl#mkitti/hdf5_1_14_compat`
[0234f1f7] HDF5_jll v1.14.0+0 `~/.julia/dev/HDF5_jll`
Oh right I fixed that on another branch: https://github.com/JuliaIO/HDF5.jl/pull/1031/files#diff-f805945337fe1c48cc5dc2d85b16017319d2e49410ebed34e70ada93924ed63f
Sorry I am a bit confused now. That other branch is missing the other fix I think ;)
Anyways, I applied the one-line patch to modify the Project.toml
of HDF5.jl
to allow for HDF5_jll@1.14.0
.
I got a few errors:
[ Info: libhdf5 v1.14.0
┌ Warning: `filter=(307, 0)` keyword option is deprecated, use `filters=(307, 0)` instead
│ caller = setproperty!(p::HDF5.DatasetCreateProperties, name::Symbol, val::Tuple{Int32, Int64}) at properties.jl:59
└ @ HDF5 ~/tmp/hdf5/pkg2/HDF5.jl/src/properties.jl:59
┌ Warning: `append!(filters::Filters.FilterPipeline, extra::NTuple{N, Integer}) where N` is deprecated, use `append!(filters, [ExternalFilter(extra...)])` instead.
│ caller = set_filters! at properties.jl:497 [inlined]
└ @ Core ~/tmp/hdf5/pkg2/HDF5.jl/src/properties.jl:497
filter: Error During Test at /Users/tamasgal/tmp/hdf5/pkg2/HDF5.jl/test/filter.jl:248
Unexpected Pass
Expression: HDF5.API.h5z_filter_avail(HDF5.API.H5Z_FILTER_SZIP)
Got correct result, please change to @test if no longer broken.
Raw Chunk I/O: Test Failed at /Users/tamasgal/tmp/hdf5/pkg2/HDF5.jl/test/chunkstorage.jl:203
Expression: iter_time < index_time
Evaluated: 0.017930666 < 0.002018916
Stacktrace:
[1] macro expansion
@ ~/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/share/julia/stdlib/v1.8/Test/src/Test.jl:464 [inlined]
[2] (::var"#352#367")(f::HDF5.File)
@ Main ~/tmp/hdf5/pkg2/HDF5.jl/test/chunkstorage.jl:203
Test Summary: | Pass Fail Error Broken Total Time
HDF5.jl | 1132 1 1 3 1137 40.9s
plain | 151 1 152 7.7s
complex | 13 13 0.3s
undefined and null | 4 4 0.0s
abstract arrays | 2 2 0.2s
empty and 0-size arrays | 39 39 0.2s
generic read of native types | 17 17 0.2s
show | 51 51 0.9s
split1 | 13 13 0.1s
haskey | 18 18 0.1s
AbstractString | 51 51 1.3s
opaque data | 7 7 0.4s
FixedStrings and FixedArrays | 18 18 0.3s
Object Exists | 6 6 0.0s
HDF5 existance | 4 4 0.0s
bounds | 2 2 0.1s
h5a_iterate | 8 8 0.1s
h5l_iterate | 8 8 0.1s
h5dchunk_iter | 3 3 0.0s
compound | 10 10 1.1s
write_compound | 18 18 0.6s
custom | 6 6 0.2s
reference | 6 6 0.1s
Dataspaces | 91 91 0.1s
Datatypes | 15 15 0.0s
BlockRange | 35 35 0.1s
hyperslab | 5 5 0.4s
read 0-length arrays: issue #859 | None 0.1s
attrs interface | 92 92 0.4s
readremote | 23 23 0.5s
extend | 29 29 0.5s
gc | 101 101 1.1s
external | 6 6 0.0s
swmr | 4 4 2.4s
mmap | 9 9 0.4s
properties | 43 1 44 0.7s
filter | 64 1 65 6.2s
bitshuffle_lz4 | 2 2 0.1s
bitshuffle_plain | 2 2 0.0s
bitshuffle_zstd | 2 2 0.0s
blosc | 3 3 0.0s
blosc_bitshuffle | 3 3 0.0s
bzip2 | 3 3 0.3s
deflate | 2 2 0.0s
filtdef | 2 2 0.0s
filtshufdef | 2 2 0.0s
lz4 | 3 3 0.0s
shufdef | 2 2 0.0s
shuffle+blosc | 4 4 0.0s
shuffle+bzip2 | 4 4 0.3s
shuffle+lz4 | 4 4 0.0s
shuffle+zstd | 4 4 0.0s
zstd | 3 3 0.0s
Raw Chunk I/O | 59 1 60 2.0s
fileio | 6 6 0.6s
track order | 18 18 0.8s
h5f_get_dset_no_attrs_hint | 6 6 0.1s
non-allocating methods | 11 1 12 0.4s
Compression Filter Unit Tests | 6 6 0.1s
Object API | 38 38 0.2s
virtual dataset | 5 5 0.3s
mpio | 1 1 0.2s
ros3 | 2 2 2.5s
ERROR: LoadError: Some tests did not pass: 1132 passed, 1 failed, 1 errored, 3 broken.
in expression starting at /Users/tamasgal/tmp/hdf5/pkg2/HDF5.jl/test/runtests.jl:18
ERROR: Package HDF5 errored during testing
Yea, I just need to collect the changes.
Otherwise, it appears to be working from what you are showing me?
Right now I'm trying to work on OpenSSL v3 issues in the BB ecosystem.
This was superseded by cross-compiled source builds in https://github.com/JuliaPackaging/Yggdrasil/pull/6551. HDF5_jll v0.14.0 is now available.
I have prepared HDF5 1.14.0 via https://github.com/JuliaPackaging/Yggdrasil/pull/6274 . Please help test this.