Profile inference for DiskArrays indexing and other tasks.

rafaqz commented 5 months ago

Profiling Rasters.jl with the NCDatasets.jl backend I noticed getindex in DiskArrays.jl is a large fraction of inference, the two large columns to the right and a lot of the area around them here:

2024-03-19-183205_1920x1080_013

It could be good to profile the first-run type inference and look at how to simplify it.

meggart commented 4 months ago

Do you happen to have the code around to reproduce that plot? Is it part of some NCDatasets tests?

rafaqz commented 4 months ago

Ugg not exactly it was Rasters.jl on main and its on a rebase branch of your NCDatasets.jl fork but just using your branch and DiskArrays 0.4 should be enough.

Heres a demo without Rasters but recreating a Rasters like workload:

using NCDatasets, ProfileView, SnoopCompile
ncfile = "tos_O1_2001-2002.nc"
# download("https://www.unidata.ucar.edu/software/netcdf/examples/$ncfile")

# @time Array(NCDatasets.Dataset(ncfile)[:tos].var .* 100)
  # 2.202080 seconds (4.64 M allocations: 321.724 MiB, 3.73% gc time, 99.69% compilation time)

# Profile compilation
@profview 1 + 1 # warmup for ProfileView
@profview Array(NCDatasets.Dataset(ncfile)[:tos].var .* 100)

# Snoop compiation
# Restart julia and don't run the profview above
tinf = @snoopi_deep Array(NCDatasets.Dataset(ncfile)[:tos].var .* 100);
fg = flamegraph(tinf)
ProfileView.view(fg)

meggart / DiskArrays.jl

Profile inference for DiskArrays indexing and other tasks. #163