search

CliMA / ClimaDiagnostics.jl

A framework to define and output observables and statistics from CliMA simulations
Apache License 2.0
7 stars 1 forks source link

`orchestrate_diagnostics` has lots of inference failures #82

Open charleskawczynski opened 1 day ago

charleskawczynski commented 1 day ago

orchestrate_diagnostics has lots of inference failures, and I've noticed that compilation time is significantly reduced when diagnostics are disabled. This very well may be the reason.

Found in https://buildkite.com/clima/climaatmos-ci/builds/20805#01922a42-bcbe-49b0-ae2d-2247cb3b2a49


││││││││││┌ orchestrate_diagnostics(integrator::ClimaTimeSteppers.DistributedODEIntegrator{…}, diagnostic_handler::ClimaDiagnostics.DiagnosticsHandler{…}) @ ClimaDiagnostics /central/scratch/esm/slurm-buildkite/climaatmos-ci/depot/default/packages/ClimaDiagnostics/Qy3We/src/clima_diagnostics.jl:195
  | │││││││││││┌ interpolate_field!(writer::ClimaDiagnostics.Writers.NetCDFWriter{…}, field::Any, diagnostic::ClimaDiagnostics.ScheduledDiagnostics.ScheduledDiagnostic{…}, u::ClimaCore.Fields.FieldVector{…}, p::ClimaAtmos.AtmosCache{…}, t::Float32) @ ClimaDiagnostics.Writers /central/scratch/esm/slurm-buildkite/climaatmos-ci/depot/default/packages/ClimaDiagnostics/Qy3We/src/netcdf_writer.jl:220
  | ││││││││││││┌ target_coordinates(space::ClimaCore.Spaces.ExtrudedFiniteDifferenceSpace, num_points::Tuple{…}, z_sampling_method::ClimaDiagnostics.Writers.LevelsMethod) @ ClimaDiagnostics.Writers /central/scratch/esm/slurm-buildkite/climaatmos-ci/depot/default/packages/ClimaDiagnostics/Qy3We/src/netcdf_writer_coordinates.jl:494
  | │││││││││││││┌  @ ClimaCore.Spaces /central/scratch/esm/slurm-buildkite/climaatmos-ci/depot/default/packages/ClimaCore/wHC4M/src/Spaces/finitedifference.jl:19
  | ││││││││││││││┌ ClimaCore.Grids.FiniteDifferenceGrid(topology::ClimaCore.Topologies.IntervalTopology) @ ClimaCore.Grids /central/scratch/esm/slurm-buildkite/climaatmos-ci/depot/default/packages/ClimaCore/wHC4M/src/Grids/finitedifference.jl:46
  | │││││││││││││││┌ get!(default::ClimaCore.Grids.var"#1#2"{…} where {…}, h::Dict{…}, key::Tuple{…}) @ Base ./dict.jl:479
  | ││││││││││││││││┌ (::ClimaCore.Grids.var"#1#2"{…} where {…})() @ ClimaCore.Grids /central/scratch/esm/slurm-buildkite/climaatmos-ci/depot/default/packages/ClimaCore/wHC4M/src/Grids/finitedifference.jl:47
  | │││││││││││││││││┌ _FiniteDifferenceGrid(topology::ClimaCore.Topologies.IntervalTopology) @ ClimaCore.Grids /central/scratch/esm/slurm-buildkite/climaatmos-ci/depot/default/packages/ClimaCore/wHC4M/src/Grids/finitedifference.jl:60
  | ││││││││││││││││││┌ (::Type{ClimaCore.DataLayouts.VF{_A, _B}} where {_A, _B})(::Type{Array{_A}} where _A, nelements::Any) @ ClimaCore.DataLayouts /central/scratch/esm/slurm-buildkite/climaatmos-ci/depot/default/packages/ClimaCore/wHC4M/src/DataLayouts/DataLayouts.jl:661
  | │││││││││││││││││││┌ (::Type{ClimaCore.DataLayouts.VF{_A, _B}} where {_A, _B})(array::Matrix) @ ClimaCore.DataLayouts /central/scratch/esm/slurm-buildkite/climaatmos-ci/depot/default/packages/ClimaCore/wHC4M/src/DataLayouts/DataLayouts.jl:649
  | ││││││││││││││││││││┌ (::Type{ClimaCore.DataLayouts.VF{_A, _B, Matrix{_A1}}} where {_A, _B, _A1})(array::Matrix) @ ClimaCore.DataLayouts /central/scratch/esm/slurm-buildkite/climaatmos-ci/depot/default/packages/ClimaCore/wHC4M/src/DataLayouts/DataLayouts.jl:642
  | │││││││││││││││││││││┌ convert(::Type{Array{NCDatasets.nc_vlen_t{T}, 2}}, data::Array{Vector{T}, 2}) where T @ NCDatasets /central/scratch/esm/slurm-buildkite/climaatmos-ci/depot/default/packages/NCDatasets/PkTUf/src/netcdf_c.jl:235
  | ││││││││││││││││││││││┌ iterate(e::Base.Iterators.Enumerate{I} where I<:(Array{Vector{T}, 2} where T)) @ Base.Iterators ./iterators.jl:205
  | │││││││││││││││││││││││┌ iterate(e::Base.Iterators.Enumerate{I} where I<:(Array{Vector{T}, 2} where T), state::Tuple{Int64}) @ Base.Iterators ./iterators.jl:206
  | ││││││││││││││││││││││││┌ iterate(A::Array{Vector{T}, 2} where T) @ Base ./array.jl:945
  | │││││││││││││││││││││││││┌ iterate(A::Array{Vector{T}, 2} where T, i::Int64) @ Base ./array.jl:945
  | ││││││││││││││││││││││││││ runtime dispatch detected: (A::Array{Vector{T}, 2} where T)[i::Int64]::Vector
  | │││││││││││││││││││││││││└────────────────────
  | │││││││││││││││││┌ _FiniteDifferenceGrid(topology::ClimaCore.Topologies.IntervalTopology) @ ClimaCore.Grids /central/scratch/esm/slurm-buildkite/climaatmos-ci/depot/default/packages/ClimaCore/wHC4M/src/Grids/finitedifference.jl:62
  | ││││││││││││││││││┌ vindex(v::Any) @ ClimaCore.DataLayouts /central/scratch/esm/slurm-buildkite/climaatmos-ci/depot/default/packages/ClimaCore/wHC4M/src/DataLayouts/DataLayouts.jl:1112
  | │││││││││││││││││││┌ CartesianIndex(::Int64, ::Int64, ::Int64, ::Integer, ::Int64) @ Base.IteratorsMD ./multidimensional.jl:79
  | ││││││││││││││││││││ runtime dispatch detected: CartesianIndex(index::Tuple{Int64, Int64, Int64, Integer, Int64})::CartesianIndex{5}
  | │││││││││││││││││││└────────────────────
  | │││││││││││││││││││┌ CartesianIndex(::Int64, ::Int64, ::Int64, ::Union{Integer, CartesianIndex}, ::Int64) @ Base.IteratorsMD ./multidimensional.jl:86
  | ││││││││││││││││││││┌ flatten(I::Tuple{Int64, Int64, Int64, Union{Integer, CartesianIndex}, Int64}) @ Base.IteratorsMD ./multidimensional.jl:89
  | │││││││││││││││││││││┌ flatten(I::Tuple{Int64, Int64, Any, Int64}) @ Base.IteratorsMD ./multidimensional.jl:89
  | ││││││││││││││││││││││┌ flatten(I::Tuple{Int64, Any, Int64}) @ Base.IteratorsMD ./multidimensional.jl:89
  | │││││││││││││││││││││││┌ flatten(I::Tuple{Any, Int64}) @ Base.IteratorsMD ./multidimensional.jl:89
  | ││││││││││││││││││││││││ runtime dispatch detected: Base.IteratorsMD.Tuple(%1::Any)::Tuple
  | │││││││││││││││││││││││└────────────────────
  | ││││││││││││││││││││┌ CartesianIndex(index::Tuple{Int64, Int64, Int64, Vararg{Union{Integer, CartesianIndex}}}) @ Base.IteratorsMD ./multidimensional.jl:90
  | │││││││││││││││││││││┌ CartesianIndex(::Int64, ::Int64, ::Int64, ::Vararg{Integer}) @ Base.IteratorsMD ./multidimensional.jl:79
  | ││││││││││││││││││││││ runtime dispatch detected: CartesianIndex(index::Tuple{Int64, Int64, Int64, Vararg{Integer}})::CartesianIndex
  | │││││││││││││││││││││└────────────────────
  | │││││││││││││││││││││┌ CartesianIndex(::Int64, ::Vararg{Union{Integer, CartesianIndex}}) @ Base.IteratorsMD ./multidimensional.jl:86
  | ││││││││││││││││││││││┌ flatten(I::Tuple{Int64, Vararg{Union{Integer, CartesianIndex}}}) @ Base.IteratorsMD ./multidimensional.jl:89
  | │││││││││││││││││││││││┌ flatten(I::Tuple{Vararg{Union{Integer, CartesianIndex}}}) @ Base.IteratorsMD ./multidimensional.jl:89
  | ││││││││││││││││││││││││ runtime dispatch detected: Base.IteratorsMD.Tuple(%1::Union{Integer, CartesianIndex})::Tuple
  | │││││││││││││││││││││││└────────────────────
  | │││││││││││││││││││││││┌ flatten(I::Tuple{Vararg{Union{Integer, CartesianIndex}}}) @ Base.IteratorsMD ./multidimensional.jl:89
  | ││││││││││││││││││││││││ runtime dispatch detected: Base.IteratorsMD.flatten(%14::Tuple{Vararg{Union{Integer, CartesianIndex}}})::Tuple
  | │││││││││││││││││││││││└────────────────────
  | ││││││││││││││││││││││┌ flatten(I::Tuple{Int64, Vararg{Union{Integer, CartesianIndex}}}) @ Base.IteratorsMD ./multidimensional.jl:89
  | │││││││││││││││││││││││ runtime dispatch detected: Base.IteratorsMD.flatten(%3::Tuple{Vararg{Union{Integer, CartesianIndex}}})::Tuple
  | ││││││││││││││││││││││└────────────────────
  | │││││││││││││││││││││┌ CartesianIndex(::Int64, ::Vararg{Union{Integer, CartesianIndex}}) @ Base.IteratorsMD ./multidimensional.jl:86
  | ││││││││││││││││││││││ failed to optimize due to recursion: CartesianIndex(::Int64, ::Vararg{Union{Integer, CartesianIndex}})
  | │││││││││││││││││││││└────────────────────
  | │││││││││││││││││││┌ CartesianIndex(::Int64, ::Int64, ::Int64, ::Union{Integer, CartesianIndex}, ::Int64) @ Base.IteratorsMD ./multidimensional.jl:86
  | ││││││││││││││││││││ failed to optimize due to recursion: CartesianIndex(::Int64, ::Int64, ::Int64, ::Union{Integer, CartesianIndex}, ::Int64)
  | │││││││││││││││││││└────────────────────
  | ││││││││││││││││││┌ vindex(v::Any) @ ClimaCore.DataLayouts /central/scratch/esm/slurm-buildkite/climaatmos-ci/depot/default/packages/ClimaCore/wHC4M/src/DataLayouts/DataLayouts.jl:1112
  | │││││││││││││││││││ runtime dispatch detected: ClimaCore.DataLayouts.CartesianIndex(1, 1, 1, v::Any, 1)::Any
  | ││││││││││││││││││└────────────────────
  | ││││││││││││││││││┌ setindex!(data::ClimaCore.DataLayouts.VF{_A, _B, Matrix{_A1}} where {_A, _B, _A1}, val::Any, I::CartesianIndex) @ ClimaCore.DataLayouts /central/scratch/esm/slurm-buildkite/climaatmos-ci/depot/default/packages/ClimaCore/wHC4M/src/DataLayouts/DataLayouts.jl:1164
  | │││││││││││││││││││┌ bounds_condition(data::ClimaCore.DataLayouts.VF{_A, _B, Matrix{_A1}} where {_A, _B, _A1}, I::CartesianIndex) @ ClimaCore.DataLayouts /central/scratch/esm/slurm-buildkite/climaatmos-ci/depot/default/packages/ClimaCore/wHC4M/src/DataLayouts/DataLayouts.jl:1009
  | ││││││││││││││││││││ runtime dispatch detected: (%2::Int64 ClimaCore.DataLayouts.:<= %7::Any)::Any
  | │││││││││││││││││││└────────────────────
  | ││││││││││││││││││┌ setindex!(data::ClimaCore.DataLayouts.VF{_A, _B, Matrix{_A1}} where {_A, _B, _A1}, val::Any, I::CartesianIndex) @ ClimaCore.DataLayouts /central/scratch/esm/slurm-buildkite/climaatmos-ci/depot/default/packages/ClimaCore/wHC4M/src/DataLayouts/DataLayouts.jl:1165
  | │││││││││││││││││││┌ set_struct!(array::Matrix{S}, val::Any, ::Val{2}, index::CartesianIndex{2}) where S @ ClimaCore.DataLayouts /central/scratch/esm/slurm-buildkite/climaatmos-ci/depot/default/packages/ClimaCore/wHC4M/src/DataLayouts/struct.jl:257
  | ││││││││││││││││││││┌ setindex!(::Matrix, ::Any, ::CartesianIndex{2}) @ Base ./multidimensional.jl:698
  | │││││││││││││││││││││┌ to_indices(A::Matrix, I::Tuple{CartesianIndex{2}}) @ Base ./multidimensional.jl:859
  | ││││││││││││││││││││││┌ to_indices(A::Matrix, inds::Tuple{}, I::Tuple{CartesianIndex{2}}) @ Base ./indices.jl:354
  | │││││││││││││││││││││││ runtime dispatch detected: Base._to_indices1(A::Matrix, inds::Tuple{}, %1::CartesianIndex{2})::Tuple{Int64, Int64}
  | ││││││││││││││││││││││└────────────────────
 ```

I had to truncate the message, since it was too long. I'm not sure how many of these are directly the result of the issue that is fixed in https://github.com/CliMA/ClimaCore.jl/pull/2004, but I can retry https://buildkite.com/clima/climaatmos-ci/builds/20805#01922a42-bcbe-49b0-ae2d-2247cb3b2a49 once that is merged/released.
charleskawczynski commented 1 day ago

I noticed that one of the failures in https://buildkite.com/clima/climaatmos-ci/builds/20805#01922a42-bcbe-49b0-ae2d-2247cb3b2a49 was an inference failure in https://github.com/CliMA/ClimaDiagnostics.jl/blob/c1cc7c38a6b9e8b18bab38babc282afc0d39d26f/src/clima_diagnostics.jl#L252-L253, which looks like it captures diagnostics_handler.