meggart / DiskArrays.jl

Other
75 stars 14 forks source link

Compare chunks of GridChunks types directly #91

Closed felixcremer closed 1 year ago

felixcremer commented 1 year ago

This avoids a costly fallback to the generic array comparison. This gives me a speedup of a YAXArrays.concatenatecubes with roughly six thousand elements of over 100x.

Before this change the comparison of two Gridchunks looked like that:

julia> @time c1 == c2
  0.021512 seconds (108.02 k allocations: 7.554 MiB)
true

with this change we got this down to zero allocations and it takes

julia> @btime c1 == c2
  13.847 μs (0 allocations: 0 bytes)
true

The real world impact is going from this:

julia> @time yax = concatenatecubes(yaxlist, taxis)
 95.495361 seconds (620.99 M allocations: 42.401 GiB, 16.22% gc time)
YAXArray with the following dimensions
Y                   Axis with 15000 Elements from 4.8e6 to 5.09998e6
X                   Axis with 15000 Elements from 2.1e6 to 1.80002e6
Band                Axis with 1 Elements from Gray to Gray
Time                Axis with 5747 Elements from 2015-01-02T05:33:02 to 2022-09-18T17:16:56
Total size: 2.35 TB

to this:

julia> @time yax = concatenatecubes(yaxlist, taxis)
  0.197601 seconds (321.97 k allocations: 15.311 MiB)
YAXArray with the following dimensions
Y                   Axis with 15000 Elements from 4.8e6 to 5.09998e6
X                   Axis with 15000 Elements from 2.1e6 to 1.80002e6
Band                Axis with 1 Elements from Gray to Gray
Time                Axis with 5747 Elements from 2015-01-02T05:33:02 to 2022-09-18T17:16:56
Total size: 2.35 TB