that's a 10-fold difference for the above arrays which are small and in memory. but even for a 450MB on-disk zarr array, yax is still 20% slower than dd:
julia> using Zarr
julia> yax = Cube("foo.zarr");
julia> dd = yaxconvert(DimArray, yax);
julia> @benchmark collect(yax[Dim{:LI}(At("bar"))])
BenchmarkTools.Trial: 73 samples with 1 evaluation.
Range (min … max): 52.840 ms … 124.095 ms ┊ GC (min … max): 3.18% … 58.09%
Time (median): 54.923 ms ┊ GC (median): 5.83%
Time (mean ± σ): 68.719 ms ± 25.640 ms ┊ GC (mean ± σ): 24.69% ± 20.59%
▂█
██▇▄▁▁▅▃▃▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▃▃▃▃▃▃▁▄▃▃ ▁
52.8 ms Histogram: frequency by time 121 ms <
Memory estimate: 126.95 MiB, allocs estimate: 10584.
julia> @benchmark collect(dd[Dim{:LI}(At("bar"))])
BenchmarkTools.Trial: 110 samples with 1 evaluation.
Range (min … max): 44.108 ms … 107.490 ms ┊ GC (min … max): 0.00% … 58.55%
Time (median): 45.175 ms ┊ GC (median): 1.25%
Time (mean ± σ): 45.998 ms ± 6.025 ms ┊ GC (mean ± σ): 2.60% ± 5.66%
▄█
▅▄▆▃▆▄█████▃▃▁▃▃▄▄▃▃▃▁▁▁▃▁▁▁▃▃▁▃▁▁▁▁▁▁▁▁▁▁▁▃▁▁▃▁▁▁▁▁▁▁▁▁▁▁▁▃ ▃
44.1 ms Histogram: frequency by time 51.9 ms <
Memory estimate: 38.41 MiB, allocs estimate: 2969.
julia> size(yax)
(20222, 1098, 145)
is this expected?
julia> versioninfo()
Julia Version 1.10.0
Commit 3120989f39b (2023-12-25 18:01 UTC)
Build Info:
Official https://julialang.org/ release
Platform Info:
OS: macOS (arm64-apple-darwin22.4.0)
CPU: 12 × Apple M2 Max
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-15.0.7 (ORCJIT, apple-m1)
Threads: 1 on 8 virtual cores
Environment:
JULIA_PROJECT = @.
JULIA_EDITOR = vi
it's slower when converting yax to dd:
as well as converting dd to yax:
that's a 10-fold difference for the above arrays which are small and in memory. but even for a 450MB on-disk zarr array, yax is still 20% slower than dd:
is this expected?
DimensionalData v0.25.8 and YAXArrays v0.5.2