Testing GPU compatible SF in ClimaAtmos

akshaysridhar commented 1 year ago

This issue tracks outcomes from a series of WIP branches (most of which have corresponding PRs) related to GPU compatibility in ClimaAtmos. Testing on NVIDIA A100-SXM4-80GB.

Relevant branches

ClimaAtmos main
SurfaceFluxes v0.8.0
ClimaAtmos as/up-sf-fluxes.

### Tasks
- [x] Demo simple configurations with Monin Obukhov fluxes (interface tests), using baroclinic wave and Held-Suarez cases.
- [x] Demo aquaplanet configuration with "DefaultExchangeCoefficients"
- [x] Demo aquaplanet configuration with "DefaultMoninObukhov"

cc @sriharshakandala

akshaysridhar commented 1 year ago

AP Test Case (Fail) : Stacktrace

asridhar@clima ~/Codes/ClimaAtmos.jl (as/up-sf-fluxes)$ julia --project=examples examples/hybrid/driver.jl --config config/longrun_configs/longrun_aquaplanet_rhoe_equil_clearsky_highres_hightop_rayleigh35e3_ft64.yml
[ Info: Running on CUDADevice.
[ Info: Setting up single-process ClimaAtmos run
[ Info: Running ClimaCore in unthreaded mode.
┌ Info: AtmosModel:
│
│                    `model_config`::`ClimaAtmos.SphericalModel`
│                       `perf_mode`::`ClimaAtmos.PerfStandard`
│                  `moisture_model`::`ClimaAtmos.EquilMoistModel`
│                     `energy_form`::`ClimaAtmos.TotalEnergy`
│                    `precip_model`::`ClimaAtmos.Microphysics0Moment`
│                    `forcing_type`::`Nothing`
│                      `subsidence`::`Nothing`
│                  `radiation_mode`::`ClimaAtmos.RRTMGPInterface.ClearSkyRadiation`
│                          `ls_adv`::`Nothing`
│                   `edmf_coriolis`::`Nothing`
│                `edmfx_entr_model`::`ClimaAtmos.NoEntrainment`
│                `edmfx_detr_model`::`ClimaAtmos.NoDetrainment`
│                  `turbconv_model`::`Nothing`
│     `non_orographic_gravity_wave`::`Nothing`
│         `orographic_gravity_wave`::`Nothing`
│                       `hyperdiff`::`ClimaAtmos.ClimaHyperdiffusion{Float32}`
│                       `vert_diff`::`ClimaAtmos.VerticalDiffusion{true, Float32}`
│                  `viscous_sponge`::`Nothing`
│                 `rayleigh_sponge`::`ClimaAtmos.RayleighSponge{Float32}`
│                 `sfc_temperature`::`ClimaAtmos.ZonallySymmetricSST`
└                   `surface_model`::`ClimaAtmos.PrescribedSurfaceTemperature`
┌ Info: numerics
│   energy_upwinding = Val{:none}()
│   tracer_upwinding = Val{:none}()
│   density_upwinding = Val{:none}()
│   edmfx_upwinding = Val{:none}()
│   apply_limiter = false
└   bubble = true
┌ Info: Time info:
│   dt = "150secs"
│   t_end = "300days"
└   floor_n_steps = 172800
┌ Info: Topography
└   topography = "NoWarp"
┌ Info: z heights
│   z_mesh.faces =
│    51-element Vector{ClimaCore.Geometry.ZPoint{Float32}}:
│     ZPoint(0.0)
│     ZPoint(30.0)
│     ZPoint(76.95174)
│     ZPoint(141.39532)
│     ZPoint(223.902)
│     ZPoint(325.0765)
│     ZPoint(445.55963)
│     ZPoint(586.0296)
│     ZPoint(747.2027)
│     ZPoint(929.832)
│     ZPoint(1134.7045)
│     ZPoint(1362.6383)
│     ZPoint(1614.4788)
│     ZPoint(1891.0958)
│     ⋮
│     ZPoint(20277.682)
│     ZPoint(21690.9)
│     ZPoint(23187.03)
│     ZPoint(24772.832)
│     ZPoint(26456.123)
│     ZPoint(28246.012)
│     ZPoint(30153.2)
│     ZPoint(32190.39)
│     ZPoint(34372.832)
│     ZPoint(36719.05)
│     ZPoint(39251.89)
│     ZPoint(42000.0)
└     ZPoint(45000.0)
┌ Info: Resolution stats:
│   Nq = 4
│   h_elem = 16
│   z_elem = 50
│   ncols = 24576
└   ndofs_total = 1228800
[ Info: Allocating Y: 24.993 s (44201897 allocations: 2.52 GiB)
┌ Error: @cuprint does not support values of type Nothing
└ @ CUDA ~/.julia/packages/CUDA/35NC6/src/device/intrinsics/output.jl:168
ERROR: LoadError: GPU compilation of MethodInstance for ClimaCore.DataLayouts.knl_copyto!(::ClimaCore.DataLayouts.VIJFH{Float32, 4, CUDA.CuDeviceArray{Float32, 5, 1}}, ::Base.Broadcast.Broadcasted{ClimaCore.DataLayouts.VIJFHStyle{4, CUDA.CuArray{Float32, N, CUDA.Mem.DeviceBuffer} where N}, NTuple{5, Base.OneTo{Int64}}, ClimaCore.Fields.var"#3#4"{Float32}, Tuple{Base.Broadcast.Broadcasted{ClimaCore.DataLayouts.VIJFHStyle{4, CUDA.CuArray{Float32, N, CUDA.Mem.DeviceBuffer} where N}, Nothing, Dierckx.Spline1D, Tuple{ClimaCore.DataLayouts.VIJFH{Float32, 4, CUDA.CuDeviceArray{Float32, 5, 1}}}}}}) failed
KernelError: passing and using non-bitstype argument

Argument 3 to your kernel function is of type Base.Broadcast.Broadcasted{ClimaCore.DataLayouts.VIJFHStyle{4, CUDA.CuArray{Float32, N, CUDA.Mem.DeviceBuffer} where N}, NTuple{5, Base.OneTo{Int64}}, ClimaCore.Fields.var"#3#4"{Float32}, Tuple{Base.Broadcast.Broadcasted{ClimaCore.DataLayouts.VIJFHStyle{4, CUDA.CuArray{Float32, N, CUDA.Mem.DeviceBuffer} where N}, Nothing, Dierckx.Spline1D, Tuple{ClimaCore.DataLayouts.VIJFH{Float32, 4, CUDA.CuDeviceArray{Float32, 5, 1}}}}}}, which is not isbits:
  .args is of type Tuple{Base.Broadcast.Broadcasted{ClimaCore.DataLayouts.VIJFHStyle{4, CUDA.CuArray{Float32, N, CUDA.Mem.DeviceBuffer} where N}, Nothing, Dierckx.Spline1D, Tuple{ClimaCore.DataLayouts.VIJFH{Float32, 4, CUDA.CuDeviceArray{Float32, 5, 1}}}}} which is not isbits.
    .1 is of type Base.Broadcast.Broadcasted{ClimaCore.DataLayouts.VIJFHStyle{4, CUDA.CuArray{Float32, N, CUDA.Mem.DeviceBuffer} where N}, Nothing, Dierckx.Spline1D, Tuple{ClimaCore.DataLayouts.VIJFH{Float32, 4, CUDA.CuDeviceArray{Float32, 5, 1}}}} which is not isbits.
      .f is of type Dierckx.Spline1D which is not isbits.
        .t is of type Vector{Float64} which is not isbits.
        .c is of type Vector{Float64} which is not isbits.
        .wrk is of type Vector{Float64} which is not isbits.

Stacktrace:
  [1] check_invocation(job::GPUCompiler.CompilerJob)
    @ GPUCompiler ~/.julia/packages/GPUCompiler/YO8Uj/src/validation.jl:96
  [2] macro expansion
    @ ~/.julia/packages/GPUCompiler/YO8Uj/src/driver.jl:123 [inlined]
  [3] macro expansion
    @ ~/.julia/packages/TimerOutputs/RsWnF/src/TimerOutput.jl:253 [inlined]
  [4] codegen(output::Symbol, job::GPUCompiler.CompilerJob; libraries::Bool, toplevel::Bool, optimize::Bool, cleanup::Bool, strip::Bool, validate::Bool, only_entry::Bool, parent_job::Nothing)
    @ GPUCompiler ~/.julia/packages/GPUCompiler/YO8Uj/src/driver.jl:121
  [5] codegen
    @ ~/.julia/packages/GPUCompiler/YO8Uj/src/driver.jl:110 [inlined]
  [6] compile(target::Symbol, job::GPUCompiler.CompilerJob; libraries::Bool, toplevel::Bool, optimize::Bool, cleanup::Bool, strip::Bool, validate::Bool, only_entry::Bool)
    @ GPUCompiler ~/.julia/packages/GPUCompiler/YO8Uj/src/driver.jl:106
  [7] compile
    @ ~/.julia/packages/GPUCompiler/YO8Uj/src/driver.jl:98 [inlined]
  [8] #1037
    @ ~/.julia/packages/CUDA/35NC6/src/compiler/compilation.jl:104 [inlined]
  [9] JuliaContext(f::CUDA.var"#1037#1040"{GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams}})
    @ GPUCompiler ~/.julia/packages/GPUCompiler/YO8Uj/src/driver.jl:47
 [10] compile(job::GPUCompiler.CompilerJob)
    @ CUDA ~/.julia/packages/CUDA/35NC6/src/compiler/compilation.jl:103
 [11] actual_compilation(cache::Dict{Any, CUDA.CuFunction}, src::Core.MethodInstance, world::UInt64, cfg::GPUCompiler.CompilerConfig{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams}, compiler::typeof(CUDA.compile), linker::typeof(CUDA.link))
    @ GPUCompiler ~/.julia/packages/GPUCompiler/YO8Uj/src/execution.jl:125
 [12] cached_compilation(cache::Dict{Any, CUDA.CuFunction}, src::Core.MethodInstance, cfg::GPUCompiler.CompilerConfig{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams}, compiler::Function, linker::Function)
    @ GPUCompiler ~/.julia/packages/GPUCompiler/YO8Uj/src/execution.jl:103
 [13] macro expansion
    @ ~/.julia/packages/CUDA/35NC6/src/compiler/execution.jl:318 [inlined]
 [14] macro expansion
    @ ./lock.jl:267 [inlined]
 [15] cufunction(f::typeof(ClimaCore.DataLayouts.knl_copyto!), tt::Type{Tuple{ClimaCore.DataLayouts.VIJFH{Float32, 4, CUDA.CuDeviceArray{Float32, 5, 1}}, Base.Broadcast.Broadcasted{ClimaCore.DataLayouts.VIJFHStyle{4, CUDA.CuArray{Float32, N, CUDA.Mem.DeviceBuffer} where N}, NTuple{5, Base.OneTo{Int64}}, ClimaCore.Fields.var"#3#4"{Float32}, Tuple{Base.Broadcast.Broadcasted{ClimaCore.DataLayouts.VIJFHStyle{4, CUDA.CuArray{Float32, N, CUDA.Mem.DeviceBuffer} where N}, Nothing, Dierckx.Spline1D, Tuple{ClimaCore.DataLayouts.VIJFH{Float32, 4, CUDA.CuDeviceArray{Float32, 5, 1}}}}}}}}; kwargs::Base.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:always_inline,), Tuple{Bool}}})
    @ CUDA ~/.julia/packages/CUDA/35NC6/src/compiler/execution.jl:313
 [16] macro expansion
    @ ~/.julia/packages/CUDA/35NC6/src/compiler/execution.jl:104 [inlined]
 [17] copyto!(dest::ClimaCore.DataLayouts.VIJFH{Float32, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}}, bc::Base.Broadcast.Broadcasted{ClimaCore.DataLayouts.VIJFHStyle{4, CUDA.CuArray{Float32, N, CUDA.Mem.DeviceBuffer} where N}, NTuple{5, Base.OneTo{Int64}}, ClimaCore.Fields.var"#3#4"{Float32}, Tuple{Base.Broadcast.Broadcasted{ClimaCore.DataLayouts.VIJFHStyle{4, CUDA.CuArray{Float32, N, CUDA.Mem.DeviceBuffer} where N}, Nothing, Dierckx.Spline1D, Tuple{ClimaCore.DataLayouts.VIJFH{Float32, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}}}}}})
    @ ClimaCore.DataLayouts ~/.julia/packages/ClimaCore/MGWpt/src/DataLayouts/cuda.jl:107
 [18] copyto!
    @ ~/.julia/packages/ClimaCore/MGWpt/src/Fields/broadcast.jl:152 [inlined]
 [19] copy
    @ ~/.julia/packages/ClimaCore/MGWpt/src/Fields/broadcast.jl:98 [inlined]
 [20] materialize(bc::Base.Broadcast.Broadcasted{ClimaCore.Fields.FieldStyle{ClimaCore.DataLayouts.VIJFHStyle{4, CUDA.CuArray{Float32, N, CUDA.Mem.DeviceBuffer} where N}}, Nothing, ClimaCore.Fields.var"#3#4"{Float32}, Tuple{Base.Broadcast.Broadcasted{ClimaCore.Fields.FieldStyle{ClimaCore.DataLayouts.VIJFHStyle{4, CUDA.CuArray{Float32, N, CUDA.Mem.DeviceBuffer} where N}}, Nothing, Dierckx.Spline1D, Tuple{Field{} (trunc disp)}}}})
    @ Base.Broadcast ./broadcast.jl:873
 [21] (::ClimaAtmos.var"#96#97"{ClimaAtmos.RRTMGPInterface.BestFit, ClimaAtmos.RRTMGPInterface.SameAsInterpolation, typeof(ClimaAtmos.rrtmgp_data_loader), ClimaCore.Fields.FieldVector{Float32, NamedTuple{(:c, :f), Tuple{Field{(:ρ, :uₕ, :ρe_tot, :ρq_tot)} (trunc disp), Field{(:u₃,)} (trunc disp)}}}, NamedTuple{(:is_init, :simulation, :atmos, :comms_ctx, :sfc_setup, :test, :moisture_model, :model_config, :limiter, :ᶜΦ, :ᶠgradᵥ_ᶜΦ, :ᶜρ_ref, :ᶜp_ref, :ᶜT, :ᶜf, :∂ᶜK∂ᶠu₃_data, :params, :energy_upwinding, :tracer_upwinding, :density_upwinding, :edmfx_upwinding, :do_dss, :ghost_buffer, :net_energy_flux_toa, :net_energy_flux_sfc, :env_thermo_quad, :ᶜspecific, :ᶜu, :ᶠu³, :ᶜK, :ᶜts, :ᶜp, :ᶜh_tot, :sfc_conditions, :ᶠtemp_scalar, :ᶜtemp_scalar, :ᶜtemp_scalar_2, :temp_data_level, :temp_data_level_2, :temp_data_level_3, :ᶜtemp_CT3, :ᶠtemp_CT3, :ᶠtemp_CT12, :ᶠtemp_CT12ʲs, :ᶠtemp_C123, :ᶜtemp_UVWxUVW, :ᶠtemp_UVWxUVW, :sfc_temp_C3, :ᶜ∇²u, :ᶜ∇²specific_energy, :ᶜ∇²specific_tracers, :hyperdiffusion_ghost_buffer, :ᶜ∇²uʲs), Tuple{Base.RefValue{Bool}, NamedTuple{(:comms_ctx, :is_debugging_tc, :output_dir, :restart, :job_id, :dt, :start_date, :t_end), Tuple{ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, Bool, String, Bool, String, Float32, Dates.DateTime, Float32}}, ClimaAtmos.AtmosModel{ClimaAtmos.SphericalModel, ClimaAtmos.PerfStandard, ClimaAtmos.EquilMoistModel, ClimaAtmos.TotalEnergy, ClimaAtmos.Microphysics0Moment, Nothing, Nothing, ClimaAtmos.RRTMGPInterface.ClearSkyRadiation, Nothing, Nothing, Bool, ClimaAtmos.NoEntrainment, ClimaAtmos.NoDetrainment, Bool, Bool, Bool, Nothing, Nothing, Nothing, ClimaAtmos.ClimaHyperdiffusion{Float32}, ClimaAtmos.VerticalDiffusion{true, Float32}, Nothing, ClimaAtmos.RayleighSponge{Float32}, ClimaAtmos.ZonallySymmetricSST, ClimaAtmos.PrescribedSurfaceTemperature}, ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, ClimaAtmos.SurfaceConditions.SurfaceState{Float32, ClimaAtmos.SurfaceConditions.MoninObukhov{Float32, Nothing, Nothing}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Nothing, ClimaAtmos.EquilMoistModel, ClimaAtmos.SphericalModel, Nothing, Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), ClimaAtmos.Parameters.ClimaAtmosParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}, RRTMGP.Parameters.RRTMGPParameters{Float32}, Insolation.Parameters.InsolationParameters{Float32}, CloudMicrophysics.Parameters.CloudMicrophysicsParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, ClimaAtmos.TurbulenceConvection.Parameters.TurbulenceConvectionParameters{Float32, CloudMicrophysics.Parameters.CloudMicrophysicsParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}}}, Val{:none}, Val{:none}, Val{:none}, Val{:none}, Bool, NamedTuple{(:c, :f), Tuple{ClimaCore.Spaces.DSSBuffer{NamedTuple{(:ρ, :uₕ, :ρe_tot, :ρq_tot), Tuple{Float32, ClimaCore.Geometry.Covariant12Vector{Float32}, Float32, Float32}}, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{NamedTuple{(:ρ, :uₕ, :ρe_tot, :ρq_tot), Tuple{Float32, ClimaCore.Geometry.UVVector{Float32}, Float32, Float32}}, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}, ClimaCore.Spaces.DSSBuffer{NamedTuple{(:u₃,), Tuple{ClimaCore.Geometry.Covariant3Vector{Float32}}}, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{NamedTuple{(:u₃,), Tuple{ClimaCore.Geometry.Covariant3Vector{Float32}}}, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}}}, Vector{ClimaCore.Geometry.WVector{Float32}}, Vector{ClimaCore.Geometry.WVector{Float32}}, ClimaAtmos.SGSQuadrature{3, ClimaAtmos.GaussianQuad, StaticArraysCore.SVector{3, Float32}, StaticArraysCore.SVector{3, Float32}}, Field{(:e_tot, :q_tot)} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{(:ts, :ustar, :obukhov_length, :buoyancy_flux, :ρ_flux_uₕ, :ρ_flux_h_tot, :ρ_flux_q_tot)} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), ClimaCore.DataLayouts.IJFH{Float32, 4, SubArray{Float32, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}, Tuple{Int64, Vararg{Base.Slice{Base.OneTo{Int64}}, 4}}, true}}, ClimaCore.DataLayouts.IJFH{Float32, 4, SubArray{Float32, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}, Tuple{Int64, Vararg{Base.Slice{Base.OneTo{Int64}}, 4}}, true}}, ClimaCore.DataLayouts.IJFH{Float32, 4, SubArray{Float32, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}, Tuple{Int64, Vararg{Base.Slice{Base.OneTo{Int64}}, 4}}, true}}, Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{(:q_tot,)} (trunc disp), NamedTuple{(:ᶜ∇²u, :ᶜ∇²specific_energy, :ᶜ∇²specific_tracers), Tuple{ClimaCore.Spaces.DSSBuffer{ClimaCore.Geometry.Covariant123Vector{Float32}, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{ClimaCore.Geometry.UVWVector{Float32}, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}, ClimaCore.Spaces.DSSBuffer{Float32, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{Float32, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}, ClimaCore.Spaces.DSSBuffer{NamedTuple{(:q_tot,), Tuple{Float32}}, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{NamedTuple{(:q_tot,), Tuple{Float32}}, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}}}, NamedTuple{(), Tuple{}}}}, ClimaAtmos.RRTMGPInterface.ClearSkyRadiation, RRTMGP.Parameters.RRTMGPParameters{Float32}, DataType, Bool, Bool, ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}})(input_data::NCDatasets.NCDataset{Nothing})
    @ ClimaAtmos ~/.julia/packages/ClimaAtmos/Y6nUf/src/parameterized_tendencies/radiation/radiation.jl:86
 [22] #92
    @ ~/.julia/packages/ClimaAtmos/Y6nUf/src/parameterized_tendencies/radiation/radiation_utilities.jl:34 [inlined]
 [23] NCDatasets.NCDataset(::ClimaAtmos.var"#92#93"{ClimaAtmos.var"#96#97"{ClimaAtmos.RRTMGPInterface.BestFit, ClimaAtmos.RRTMGPInterface.SameAsInterpolation, typeof(ClimaAtmos.rrtmgp_data_loader), ClimaCore.Fields.FieldVector{Float32, NamedTuple{(:c, :f), Tuple{Field{(:ρ, :uₕ, :ρe_tot, :ρq_tot)} (trunc disp), Field{(:u₃,)} (trunc disp)}}}, NamedTuple{(:is_init, :simulation, :atmos, :comms_ctx, :sfc_setup, :test, :moisture_model, :model_config, :limiter, :ᶜΦ, :ᶠgradᵥ_ᶜΦ, :ᶜρ_ref, :ᶜp_ref, :ᶜT, :ᶜf, :∂ᶜK∂ᶠu₃_data, :params, :energy_upwinding, :tracer_upwinding, :density_upwinding, :edmfx_upwinding, :do_dss, :ghost_buffer, :net_energy_flux_toa, :net_energy_flux_sfc, :env_thermo_quad, :ᶜspecific, :ᶜu, :ᶠu³, :ᶜK, :ᶜts, :ᶜp, :ᶜh_tot, :sfc_conditions, :ᶠtemp_scalar, :ᶜtemp_scalar, :ᶜtemp_scalar_2, :temp_data_level, :temp_data_level_2, :temp_data_level_3, :ᶜtemp_CT3, :ᶠtemp_CT3, :ᶠtemp_CT12, :ᶠtemp_CT12ʲs, :ᶠtemp_C123, :ᶜtemp_UVWxUVW, :ᶠtemp_UVWxUVW, :sfc_temp_C3, :ᶜ∇²u, :ᶜ∇²specific_energy, :ᶜ∇²specific_tracers, :hyperdiffusion_ghost_buffer, :ᶜ∇²uʲs), Tuple{Base.RefValue{Bool}, NamedTuple{(:comms_ctx, :is_debugging_tc, :output_dir, :restart, :job_id, :dt, :start_date, :t_end), Tuple{ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, Bool, String, Bool, String, Float32, Dates.DateTime, Float32}}, ClimaAtmos.AtmosModel{ClimaAtmos.SphericalModel, ClimaAtmos.PerfStandard, ClimaAtmos.EquilMoistModel, ClimaAtmos.TotalEnergy, ClimaAtmos.Microphysics0Moment, Nothing, Nothing, ClimaAtmos.RRTMGPInterface.ClearSkyRadiation, Nothing, Nothing, Bool, ClimaAtmos.NoEntrainment, ClimaAtmos.NoDetrainment, Bool, Bool, Bool, Nothing, Nothing, Nothing, ClimaAtmos.ClimaHyperdiffusion{Float32}, ClimaAtmos.VerticalDiffusion{true, Float32}, Nothing, ClimaAtmos.RayleighSponge{Float32}, ClimaAtmos.ZonallySymmetricSST, ClimaAtmos.PrescribedSurfaceTemperature}, ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, ClimaAtmos.SurfaceConditions.SurfaceState{Float32, ClimaAtmos.SurfaceConditions.MoninObukhov{Float32, Nothing, Nothing}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Nothing, ClimaAtmos.EquilMoistModel, ClimaAtmos.SphericalModel, Nothing, Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), ClimaAtmos.Parameters.ClimaAtmosParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}, RRTMGP.Parameters.RRTMGPParameters{Float32}, Insolation.Parameters.InsolationParameters{Float32}, CloudMicrophysics.Parameters.CloudMicrophysicsParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, ClimaAtmos.TurbulenceConvection.Parameters.TurbulenceConvectionParameters{Float32, CloudMicrophysics.Parameters.CloudMicrophysicsParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}}}, Val{:none}, Val{:none}, Val{:none}, Val{:none}, Bool, NamedTuple{(:c, :f), Tuple{ClimaCore.Spaces.DSSBuffer{NamedTuple{(:ρ, :uₕ, :ρe_tot, :ρq_tot), Tuple{Float32, ClimaCore.Geometry.Covariant12Vector{Float32}, Float32, Float32}}, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{NamedTuple{(:ρ, :uₕ, :ρe_tot, :ρq_tot), Tuple{Float32, ClimaCore.Geometry.UVVector{Float32}, Float32, Float32}}, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}, ClimaCore.Spaces.DSSBuffer{NamedTuple{(:u₃,), Tuple{ClimaCore.Geometry.Covariant3Vector{Float32}}}, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{NamedTuple{(:u₃,), Tuple{ClimaCore.Geometry.Covariant3Vector{Float32}}}, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}}}, Vector{ClimaCore.Geometry.WVector{Float32}}, Vector{ClimaCore.Geometry.WVector{Float32}}, ClimaAtmos.SGSQuadrature{3, ClimaAtmos.GaussianQuad, StaticArraysCore.SVector{3, Float32}, StaticArraysCore.SVector{3, Float32}}, Field{(:e_tot, :q_tot)} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{(:ts, :ustar, :obukhov_length, :buoyancy_flux, :ρ_flux_uₕ, :ρ_flux_h_tot, :ρ_flux_q_tot)} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), ClimaCore.DataLayouts.IJFH{Float32, 4, SubArray{Float32, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}, Tuple{Int64, Vararg{Base.Slice{Base.OneTo{Int64}}, 4}}, true}}, ClimaCore.DataLayouts.IJFH{Float32, 4, SubArray{Float32, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}, Tuple{Int64, Vararg{Base.Slice{Base.OneTo{Int64}}, 4}}, true}}, ClimaCore.DataLayouts.IJFH{Float32, 4, SubArray{Float32, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}, Tuple{Int64, Vararg{Base.Slice{Base.OneTo{Int64}}, 4}}, true}}, Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{(:q_tot,)} (trunc disp), NamedTuple{(:ᶜ∇²u, :ᶜ∇²specific_energy, :ᶜ∇²specific_tracers), Tuple{ClimaCore.Spaces.DSSBuffer{ClimaCore.Geometry.Covariant123Vector{Float32}, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{ClimaCore.Geometry.UVWVector{Float32}, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}, ClimaCore.Spaces.DSSBuffer{Float32, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{Float32, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}, ClimaCore.Spaces.DSSBuffer{NamedTuple{(:q_tot,), Tuple{Float32}}, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{NamedTuple{(:q_tot,), Tuple{Float32}}, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}}}, NamedTuple{(), Tuple{}}}}, ClimaAtmos.RRTMGPInterface.ClearSkyRadiation, RRTMGP.Parameters.RRTMGPParameters{Float32}, DataType, Bool, Bool, ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}}}, ::String, ::Vararg{String}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ NCDatasets ~/.julia/packages/NCDatasets/c8XyT/src/dataset.jl:220
 [24] NCDataset
    @ ~/.julia/packages/NCDatasets/c8XyT/src/dataset.jl:217 [inlined]
 [25] rrtmgp_data_loader
    @ ~/.julia/packages/ClimaAtmos/Y6nUf/src/parameterized_tendencies/radiation/radiation_utilities.jl:33 [inlined]
 [26] radiation_model_cache(Y::ClimaCore.Fields.FieldVector{Float32, NamedTuple{(:c, :f), Tuple{Field{(:ρ, :uₕ, :ρe_tot, :ρq_tot)} (trunc disp), Field{(:u₃,)} (trunc disp)}}}, default_cache::NamedTuple{(:is_init, :simulation, :atmos, :comms_ctx, :sfc_setup, :test, :moisture_model, :model_config, :limiter, :ᶜΦ, :ᶠgradᵥ_ᶜΦ, :ᶜρ_ref, :ᶜp_ref, :ᶜT, :ᶜf, :∂ᶜK∂ᶠu₃_data, :params, :energy_upwinding, :tracer_upwinding, :density_upwinding, :edmfx_upwinding, :do_dss, :ghost_buffer, :net_energy_flux_toa, :net_energy_flux_sfc, :env_thermo_quad, :ᶜspecific, :ᶜu, :ᶠu³, :ᶜK, :ᶜts, :ᶜp, :ᶜh_tot, :sfc_conditions, :ᶠtemp_scalar, :ᶜtemp_scalar, :ᶜtemp_scalar_2, :temp_data_level, :temp_data_level_2, :temp_data_level_3, :ᶜtemp_CT3, :ᶠtemp_CT3, :ᶠtemp_CT12, :ᶠtemp_CT12ʲs, :ᶠtemp_C123, :ᶜtemp_UVWxUVW, :ᶠtemp_UVWxUVW, :sfc_temp_C3, :ᶜ∇²u, :ᶜ∇²specific_energy, :ᶜ∇²specific_tracers, :hyperdiffusion_ghost_buffer, :ᶜ∇²uʲs), Tuple{Base.RefValue{Bool}, NamedTuple{(:comms_ctx, :is_debugging_tc, :output_dir, :restart, :job_id, :dt, :start_date, :t_end), Tuple{ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, Bool, String, Bool, String, Float32, Dates.DateTime, Float32}}, ClimaAtmos.AtmosModel{ClimaAtmos.SphericalModel, ClimaAtmos.PerfStandard, ClimaAtmos.EquilMoistModel, ClimaAtmos.TotalEnergy, ClimaAtmos.Microphysics0Moment, Nothing, Nothing, ClimaAtmos.RRTMGPInterface.ClearSkyRadiation, Nothing, Nothing, Bool, ClimaAtmos.NoEntrainment, ClimaAtmos.NoDetrainment, Bool, Bool, Bool, Nothing, Nothing, Nothing, ClimaAtmos.ClimaHyperdiffusion{Float32}, ClimaAtmos.VerticalDiffusion{true, Float32}, Nothing, ClimaAtmos.RayleighSponge{Float32}, ClimaAtmos.ZonallySymmetricSST, ClimaAtmos.PrescribedSurfaceTemperature}, ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, ClimaAtmos.SurfaceConditions.SurfaceState{Float32, ClimaAtmos.SurfaceConditions.MoninObukhov{Float32, Nothing, Nothing}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Nothing, ClimaAtmos.EquilMoistModel, ClimaAtmos.SphericalModel, Nothing, Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), ClimaAtmos.Parameters.ClimaAtmosParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}, RRTMGP.Parameters.RRTMGPParameters{Float32}, Insolation.Parameters.InsolationParameters{Float32}, CloudMicrophysics.Parameters.CloudMicrophysicsParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, ClimaAtmos.TurbulenceConvection.Parameters.TurbulenceConvectionParameters{Float32, CloudMicrophysics.Parameters.CloudMicrophysicsParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}}}, Val{:none}, Val{:none}, Val{:none}, Val{:none}, Bool, NamedTuple{(:c, :f), Tuple{ClimaCore.Spaces.DSSBuffer{NamedTuple{(:ρ, :uₕ, :ρe_tot, :ρq_tot), Tuple{Float32, ClimaCore.Geometry.Covariant12Vector{Float32}, Float32, Float32}}, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{NamedTuple{(:ρ, :uₕ, :ρe_tot, :ρq_tot), Tuple{Float32, ClimaCore.Geometry.UVVector{Float32}, Float32, Float32}}, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}, ClimaCore.Spaces.DSSBuffer{NamedTuple{(:u₃,), Tuple{ClimaCore.Geometry.Covariant3Vector{Float32}}}, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{NamedTuple{(:u₃,), Tuple{ClimaCore.Geometry.Covariant3Vector{Float32}}}, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}}}, Vector{ClimaCore.Geometry.WVector{Float32}}, Vector{ClimaCore.Geometry.WVector{Float32}}, ClimaAtmos.SGSQuadrature{3, ClimaAtmos.GaussianQuad, StaticArraysCore.SVector{3, Float32}, StaticArraysCore.SVector{3, Float32}}, Field{(:e_tot, :q_tot)} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{(:ts, :ustar, :obukhov_length, :buoyancy_flux, :ρ_flux_uₕ, :ρ_flux_h_tot, :ρ_flux_q_tot)} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), ClimaCore.DataLayouts.IJFH{Float32, 4, SubArray{Float32, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}, Tuple{Int64, Vararg{Base.Slice{Base.OneTo{Int64}}, 4}}, true}}, ClimaCore.DataLayouts.IJFH{Float32, 4, SubArray{Float32, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}, Tuple{Int64, Vararg{Base.Slice{Base.OneTo{Int64}}, 4}}, true}}, ClimaCore.DataLayouts.IJFH{Float32, 4, SubArray{Float32, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}, Tuple{Int64, Vararg{Base.Slice{Base.OneTo{Int64}}, 4}}, true}}, Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{(:q_tot,)} (trunc disp), NamedTuple{(:ᶜ∇²u, :ᶜ∇²specific_energy, :ᶜ∇²specific_tracers), Tuple{ClimaCore.Spaces.DSSBuffer{ClimaCore.Geometry.Covariant123Vector{Float32}, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{ClimaCore.Geometry.UVWVector{Float32}, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}, ClimaCore.Spaces.DSSBuffer{Float32, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{Float32, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}, ClimaCore.Spaces.DSSBuffer{NamedTuple{(:q_tot,), Tuple{Float32}}, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{NamedTuple{(:q_tot,), Tuple{Float32}}, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}}}, NamedTuple{(), Tuple{}}}}, params::ClimaAtmos.Parameters.ClimaAtmosParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}, RRTMGP.Parameters.RRTMGPParameters{Float32}, Insolation.Parameters.InsolationParameters{Float32}, CloudMicrophysics.Parameters.CloudMicrophysicsParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, ClimaAtmos.TurbulenceConvection.Parameters.TurbulenceConvectionParameters{Float32, CloudMicrophysics.Parameters.CloudMicrophysicsParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}}}, radiation_mode::ClimaAtmos.RRTMGPInterface.ClearSkyRadiation; interpolation::ClimaAtmos.RRTMGPInterface.BestFit, bottom_extrapolation::ClimaAtmos.RRTMGPInterface.SameAsInterpolation, data_loader::typeof(ClimaAtmos.rrtmgp_data_loader))
    @ ClimaAtmos ~/.julia/packages/ClimaAtmos/Y6nUf/src/parameterized_tendencies/radiation/radiation.jl:60
 [27] radiation_model_cache
    @ ~/.julia/packages/ClimaAtmos/Y6nUf/src/parameterized_tendencies/radiation/radiation.jl:29 [inlined]
 [28] additional_cache(Y::ClimaCore.Fields.FieldVector{Float32, NamedTuple{(:c, :f), Tuple{Field{(:ρ, :uₕ, :ρe_tot, :ρq_tot)} (trunc disp), Field{(:u₃,)} (trunc disp)}}}, default_cache::NamedTuple{(:is_init, :simulation, :atmos, :comms_ctx, :sfc_setup, :test, :moisture_model, :model_config, :limiter, :ᶜΦ, :ᶠgradᵥ_ᶜΦ, :ᶜρ_ref, :ᶜp_ref, :ᶜT, :ᶜf, :∂ᶜK∂ᶠu₃_data, :params, :energy_upwinding, :tracer_upwinding, :density_upwinding, :edmfx_upwinding, :do_dss, :ghost_buffer, :net_energy_flux_toa, :net_energy_flux_sfc, :env_thermo_quad, :ᶜspecific, :ᶜu, :ᶠu³, :ᶜK, :ᶜts, :ᶜp, :ᶜh_tot, :sfc_conditions, :ᶠtemp_scalar, :ᶜtemp_scalar, :ᶜtemp_scalar_2, :temp_data_level, :temp_data_level_2, :temp_data_level_3, :ᶜtemp_CT3, :ᶠtemp_CT3, :ᶠtemp_CT12, :ᶠtemp_CT12ʲs, :ᶠtemp_C123, :ᶜtemp_UVWxUVW, :ᶠtemp_UVWxUVW, :sfc_temp_C3, :ᶜ∇²u, :ᶜ∇²specific_energy, :ᶜ∇²specific_tracers, :hyperdiffusion_ghost_buffer, :ᶜ∇²uʲs), Tuple{Base.RefValue{Bool}, NamedTuple{(:comms_ctx, :is_debugging_tc, :output_dir, :restart, :job_id, :dt, :start_date, :t_end), Tuple{ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, Bool, String, Bool, String, Float32, Dates.DateTime, Float32}}, ClimaAtmos.AtmosModel{ClimaAtmos.SphericalModel, ClimaAtmos.PerfStandard, ClimaAtmos.EquilMoistModel, ClimaAtmos.TotalEnergy, ClimaAtmos.Microphysics0Moment, Nothing, Nothing, ClimaAtmos.RRTMGPInterface.ClearSkyRadiation, Nothing, Nothing, Bool, ClimaAtmos.NoEntrainment, ClimaAtmos.NoDetrainment, Bool, Bool, Bool, Nothing, Nothing, Nothing, ClimaAtmos.ClimaHyperdiffusion{Float32}, ClimaAtmos.VerticalDiffusion{true, Float32}, Nothing, ClimaAtmos.RayleighSponge{Float32}, ClimaAtmos.ZonallySymmetricSST, ClimaAtmos.PrescribedSurfaceTemperature}, ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, ClimaAtmos.SurfaceConditions.SurfaceState{Float32, ClimaAtmos.SurfaceConditions.MoninObukhov{Float32, Nothing, Nothing}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Nothing, ClimaAtmos.EquilMoistModel, ClimaAtmos.SphericalModel, Nothing, Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), ClimaAtmos.Parameters.ClimaAtmosParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}, RRTMGP.Parameters.RRTMGPParameters{Float32}, Insolation.Parameters.InsolationParameters{Float32}, CloudMicrophysics.Parameters.CloudMicrophysicsParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, ClimaAtmos.TurbulenceConvection.Parameters.TurbulenceConvectionParameters{Float32, CloudMicrophysics.Parameters.CloudMicrophysicsParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}}}, Val{:none}, Val{:none}, Val{:none}, Val{:none}, Bool, NamedTuple{(:c, :f), Tuple{ClimaCore.Spaces.DSSBuffer{NamedTuple{(:ρ, :uₕ, :ρe_tot, :ρq_tot), Tuple{Float32, ClimaCore.Geometry.Covariant12Vector{Float32}, Float32, Float32}}, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{NamedTuple{(:ρ, :uₕ, :ρe_tot, :ρq_tot), Tuple{Float32, ClimaCore.Geometry.UVVector{Float32}, Float32, Float32}}, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}, ClimaCore.Spaces.DSSBuffer{NamedTuple{(:u₃,), Tuple{ClimaCore.Geometry.Covariant3Vector{Float32}}}, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{NamedTuple{(:u₃,), Tuple{ClimaCore.Geometry.Covariant3Vector{Float32}}}, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}}}, Vector{ClimaCore.Geometry.WVector{Float32}}, Vector{ClimaCore.Geometry.WVector{Float32}}, ClimaAtmos.SGSQuadrature{3, ClimaAtmos.GaussianQuad, StaticArraysCore.SVector{3, Float32}, StaticArraysCore.SVector{3, Float32}}, Field{(:e_tot, :q_tot)} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{(:ts, :ustar, :obukhov_length, :buoyancy_flux, :ρ_flux_uₕ, :ρ_flux_h_tot, :ρ_flux_q_tot)} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), ClimaCore.DataLayouts.IJFH{Float32, 4, SubArray{Float32, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}, Tuple{Int64, Vararg{Base.Slice{Base.OneTo{Int64}}, 4}}, true}}, ClimaCore.DataLayouts.IJFH{Float32, 4, SubArray{Float32, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}, Tuple{Int64, Vararg{Base.Slice{Base.OneTo{Int64}}, 4}}, true}}, ClimaCore.DataLayouts.IJFH{Float32, 4, SubArray{Float32, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}, Tuple{Int64, Vararg{Base.Slice{Base.OneTo{Int64}}, 4}}, true}}, Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{(:q_tot,)} (trunc disp), NamedTuple{(:ᶜ∇²u, :ᶜ∇²specific_energy, :ᶜ∇²specific_tracers), Tuple{ClimaCore.Spaces.DSSBuffer{ClimaCore.Geometry.Covariant123Vector{Float32}, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{ClimaCore.Geometry.UVWVector{Float32}, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}, ClimaCore.Spaces.DSSBuffer{Float32, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{Float32, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}, ClimaCore.Spaces.DSSBuffer{NamedTuple{(:q_tot,), Tuple{Float32}}, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{NamedTuple{(:q_tot,), Tuple{Float32}}, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}}}, NamedTuple{(), Tuple{}}}}, parsed_args::Dict{Any, Any}, params::ClimaAtmos.Parameters.ClimaAtmosParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}, RRTMGP.Parameters.RRTMGPParameters{Float32}, Insolation.Parameters.InsolationParameters{Float32}, CloudMicrophysics.Parameters.CloudMicrophysicsParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, ClimaAtmos.TurbulenceConvection.Parameters.TurbulenceConvectionParameters{Float32, CloudMicrophysics.Parameters.CloudMicrophysicsParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}}}, atmos::ClimaAtmos.AtmosModel{ClimaAtmos.SphericalModel, ClimaAtmos.PerfStandard, ClimaAtmos.EquilMoistModel, ClimaAtmos.TotalEnergy, ClimaAtmos.Microphysics0Moment, Nothing, Nothing, ClimaAtmos.RRTMGPInterface.ClearSkyRadiation, Nothing, Nothing, Bool, ClimaAtmos.NoEntrainment, ClimaAtmos.NoDetrainment, Bool, Bool, Bool, Nothing, Nothing, Nothing, ClimaAtmos.ClimaHyperdiffusion{Float32}, ClimaAtmos.VerticalDiffusion{true, Float32}, Nothing, ClimaAtmos.RayleighSponge{Float32}, ClimaAtmos.ZonallySymmetricSST, ClimaAtmos.PrescribedSurfaceTemperature}, dt::Float32, initial_condition::ClimaAtmos.InitialConditions.DecayingProfile)
    @ ClimaAtmos ~/.julia/packages/ClimaAtmos/Y6nUf/src/cache/cache.jl:134
 [29] get_cache(Y::ClimaCore.Fields.FieldVector{Float32, NamedTuple{(:c, :f), Tuple{Field{(:ρ, :uₕ, :ρe_tot, :ρq_tot)} (trunc disp), Field{(:u₃,)} (trunc disp)}}}, parsed_args::Dict{Any, Any}, params::ClimaAtmos.Parameters.ClimaAtmosParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}, RRTMGP.Parameters.RRTMGPParameters{Float32}, Insolation.Parameters.InsolationParameters{Float32}, CloudMicrophysics.Parameters.CloudMicrophysicsParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, ClimaAtmos.TurbulenceConvection.Parameters.TurbulenceConvectionParameters{Float32, CloudMicrophysics.Parameters.CloudMicrophysicsParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}}}, spaces::NamedTuple{(:center_space, :face_space, :horizontal_mesh, :quad, :z_max, :z_elem, :z_stretch), Tuple{ClimaCore.Spaces.ExtrudedFiniteDifferenceSpace{ClimaCore.Spaces.CellCenter, ClimaCore.Spaces.SpectralElementSpace2D{ClimaCore.Topologies.Topology2D{ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, ClimaCore.Meshes.EquiangularCubedSphere{ClimaCore.Domains.SphereDomain{Float32}, ClimaCore.Meshes.NormalizedBilinearMap}, Vector{CartesianIndex{3}}, Array{Int64, 3}, CUDA.CuArray{Tuple{Int64, Int64, Int64, Int64, Bool}, 1, CUDA.Mem.DeviceBuffer}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, CUDA.CuArray{Tuple{Int64, Int64}, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Tuple{Bool, Int64, Int64}, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, NamedTuple{(), Tuple{}}, CUDA.CuArray{Tuple{Int64, Int64}, 1, CUDA.Mem.DeviceBuffer}}, ClimaCore.Spaces.Quadratures.GLL{4}, ClimaCore.Geometry.SphericalGlobalGeometry{Float32}, ClimaCore.DataLayouts.IJFH{ClimaCore.Geometry.LocalGeometry{(1, 2), ClimaCore.Geometry.LatLongPoint{Float32}, Float32, StaticArraysCore.SMatrix{2, 2, Float32, 4}}, 4, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, ClimaCore.DataLayouts.IJFH{Float32, 4, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, ClimaCore.DataLayouts.IFH{ClimaCore.Geometry.SurfaceGeometry{Float32, ClimaCore.Geometry.UVVector{Float32}}, 4, CUDA.CuArray{Float32, 3, CUDA.Mem.DeviceBuffer}}, NamedTuple{(), Tuple{}}}, ClimaCore.Topologies.IntervalTopology{ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.ZPoint{Float32}, Tuple{Symbol, Symbol}}, Vector{ClimaCore.Geometry.ZPoint{Float32}}}, NamedTuple{(:bottom, :top), Tuple{Int64, Int64}}}, ClimaCore.Spaces.Flat, ClimaCore.Geometry.SphericalGlobalGeometry{Float32}, ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.LatLongZPoint{Float32}, Float32, StaticArraysCore.SMatrix{3, 3, Float32, 9}}, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}}, ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.LatLongZPoint{Float32}, Float32, StaticArraysCore.SMatrix{3, 3, Float32, 9}}, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}}}, ClimaCore.Spaces.ExtrudedFiniteDifferenceSpace{ClimaCore.Spaces.CellFace, ClimaCore.Spaces.SpectralElementSpace2D{ClimaCore.Topologies.Topology2D{ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, ClimaCore.Meshes.EquiangularCubedSphere{ClimaCore.Domains.SphereDomain{Float32}, ClimaCore.Meshes.NormalizedBilinearMap}, Vector{CartesianIndex{3}}, Array{Int64, 3}, CUDA.CuArray{Tuple{Int64, Int64, Int64, Int64, Bool}, 1, CUDA.Mem.DeviceBuffer}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, CUDA.CuArray{Tuple{Int64, Int64}, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Tuple{Bool, Int64, Int64}, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, NamedTuple{(), Tuple{}}, CUDA.CuArray{Tuple{Int64, Int64}, 1, CUDA.Mem.DeviceBuffer}}, ClimaCore.Spaces.Quadratures.GLL{4}, ClimaCore.Geometry.SphericalGlobalGeometry{Float32}, ClimaCore.DataLayouts.IJFH{ClimaCore.Geometry.LocalGeometry{(1, 2), ClimaCore.Geometry.LatLongPoint{Float32}, Float32, StaticArraysCore.SMatrix{2, 2, Float32, 4}}, 4, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, ClimaCore.DataLayouts.IJFH{Float32, 4, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, ClimaCore.DataLayouts.IFH{ClimaCore.Geometry.SurfaceGeometry{Float32, ClimaCore.Geometry.UVVector{Float32}}, 4, CUDA.CuArray{Float32, 3, CUDA.Mem.DeviceBuffer}}, NamedTuple{(), Tuple{}}}, ClimaCore.Topologies.IntervalTopology{ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.ZPoint{Float32}, Tuple{Symbol, Symbol}}, Vector{ClimaCore.Geometry.ZPoint{Float32}}}, NamedTuple{(:bottom, :top), Tuple{Int64, Int64}}}, ClimaCore.Spaces.Flat, ClimaCore.Geometry.SphericalGlobalGeometry{Float32}, ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.LatLongZPoint{Float32}, Float32, StaticArraysCore.SMatrix{3, 3, Float32, 9}}, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}}, ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.LatLongZPoint{Float32}, Float32, StaticArraysCore.SMatrix{3, 3, Float32, 9}}, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}}}, ClimaCore.Meshes.EquiangularCubedSphere{ClimaCore.Domains.SphereDomain{Float32}, ClimaCore.Meshes.NormalizedBilinearMap}, ClimaCore.Spaces.Quadratures.GLL{4}, Float32, Int64, ClimaCore.Meshes.GeneralizedExponentialStretching{Float32}}}, atmos::ClimaAtmos.AtmosModel{ClimaAtmos.SphericalModel, ClimaAtmos.PerfStandard, ClimaAtmos.EquilMoistModel, ClimaAtmos.TotalEnergy, ClimaAtmos.Microphysics0Moment, Nothing, Nothing, ClimaAtmos.RRTMGPInterface.ClearSkyRadiation, Nothing, Nothing, Bool, ClimaAtmos.NoEntrainment, ClimaAtmos.NoDetrainment, Bool, Bool, Bool, Nothing, Nothing, Nothing, ClimaAtmos.ClimaHyperdiffusion{Float32}, ClimaAtmos.VerticalDiffusion{true, Float32}, Nothing, ClimaAtmos.RayleighSponge{Float32}, ClimaAtmos.ZonallySymmetricSST, ClimaAtmos.PrescribedSurfaceTemperature}, numerics::NamedTuple{(:energy_upwinding, :tracer_upwinding, :density_upwinding, :edmfx_upwinding, :apply_limiter, :bubble), Tuple{Val{:none}, Val{:none}, Val{:none}, Val{:none}, Bool, Bool}}, simulation::NamedTuple{(:comms_ctx, :is_debugging_tc, :output_dir, :restart, :job_id, :dt, :start_date, :t_end), Tuple{ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, Bool, String, Bool, String, Float32, Dates.DateTime, Float32}}, initial_condition::ClimaAtmos.InitialConditions.DecayingProfile, surface_setup::ClimaAtmos.SurfaceConditions.DefaultMoninObukhov)
    @ ClimaAtmos ~/.julia/packages/ClimaAtmos/Y6nUf/src/solver/type_getters.jl:562
 [30] macro expansion
    @ ~/.julia/packages/ClimaAtmos/Y6nUf/src/solver/type_getters.jl:819 [inlined]
 [31] macro expansion
    @ ./timing.jl:501 [inlined]
 [32] macro expansion
    @ ~/.julia/packages/ClimaAtmos/Y6nUf/src/utils/utilities.jl:257 [inlined]
 [33] get_integrator(config::ClimaAtmos.AtmosConfig{Float32, CLIMAParameters.AliasParamDict{Float32}, Dict{Any, Any}, ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}})
    @ ClimaAtmos ~/.julia/packages/ClimaAtmos/Y6nUf/src/solver/type_getters.jl:818
 [34] top-level scope
    @ ~/Codes/ClimaAtmos.jl/examples/hybrid/driver.jl:8
in expression starting at /home/asridhar/Codes/ClimaAtmos.jl/examples/hybrid/driver.jl:8

akshaysridhar commented 1 year ago

Baroclinic Wave SF Interface (similar outcome for Held-Suarez w/ MO) Test (Pass)

asridhar@clima ~/Codes/ClimaAtmos.jl (as/up-sf-fluxes)$ julia --project=examples examples/hybrid/driver.jl --config config/gpu_configs/target_gpu_implicit_moist_baroclinic_wave_sfmo.yml
[ Info: Running on CUDADevice.
[ Info: Setting up single-process ClimaAtmos run
[ Info: Running ClimaCore in unthreaded mode.
┌ Info: AtmosModel:
│
│                    `model_config`::`ClimaAtmos.SphericalModel`
│                       `perf_mode`::`ClimaAtmos.PerfStandard`
│                  `moisture_model`::`ClimaAtmos.EquilMoistModel`
│                     `energy_form`::`ClimaAtmos.TotalEnergy`
│                    `precip_model`::`ClimaAtmos.Microphysics0Moment`
│                    `forcing_type`::`Nothing`
│                      `subsidence`::`Nothing`
│                  `radiation_mode`::`Nothing`
│                          `ls_adv`::`Nothing`
│                   `edmf_coriolis`::`Nothing`
│                `edmfx_entr_model`::`ClimaAtmos.NoEntrainment`
│                `edmfx_detr_model`::`ClimaAtmos.NoDetrainment`
│                  `turbconv_model`::`Nothing`
│     `non_orographic_gravity_wave`::`Nothing`
│         `orographic_gravity_wave`::`Nothing`
│                       `hyperdiff`::`ClimaAtmos.ClimaHyperdiffusion{Float32}`
│                       `vert_diff`::`Nothing`
│                  `viscous_sponge`::`Nothing`
│                 `rayleigh_sponge`::`Nothing`
│                 `sfc_temperature`::`ClimaAtmos.ZonallySymmetricSST`
└                   `surface_model`::`ClimaAtmos.PrescribedSurfaceTemperature`
┌ Info: numerics
│   energy_upwinding = Val{:none}()
│   tracer_upwinding = Val{:none}()
│   density_upwinding = Val{:none}()
│   edmfx_upwinding = Val{:none}()
│   apply_limiter = false
└   bubble = true
┌ Info: Time info:
│   dt = "100secs"
│   t_end = "15days"
└   floor_n_steps = 12960
┌ Info: Topography
└   topography = "NoWarp"
┌ Info: z heights
│   z_mesh.faces =
│    64-element Vector{ClimaCore.Geometry.ZPoint{Float32}}:
│     ZPoint(0.0)
│     ZPoint(30.0)
│     ZPoint(64.432526)
│     ZPoint(103.41123)
│     ZPoint(147.05489)
│     ZPoint(195.48775)
│     ZPoint(248.8398)
│     ZPoint(307.24713)
│     ZPoint(370.8522)
│     ZPoint(439.80435)
│     ZPoint(514.26)
│     ZPoint(594.383)
│     ZPoint(680.34534)
│     ZPoint(772.3272)
│     ⋮
│     ZPoint(12489.93)
│     ZPoint(13257.988)
│     ZPoint(14084.975)
│     ZPoint(14979.377)
│     ZPoint(15951.814)
│     ZPoint(17015.83)
│     ZPoint(18189.092)
│     ZPoint(19495.287)
│     ZPoint(20967.266)
│     ZPoint(22652.527)
│     ZPoint(24623.646)
│     ZPoint(27000.0)
└     ZPoint(30000.0)
┌ Info: Resolution stats:
│   Nq = 4
│   h_elem = 30
│   z_elem = 63
│   ncols = 86400
└   ndofs_total = 5443200
[ Info: Allocating Y: 27.883 s (56900391 allocations: 2.96 GiB)
┌ Error: @cuprint does not support values of type Nothing
└ @ CUDA ~/.julia/packages/CUDA/35NC6/src/device/intrinsics/output.jl:168
[ Info: Allocating cache (p): 36.319 s (158948327 allocations: 6.56 GiB)
[ Info: Using ODE config: `ClimaTimeSteppers.ARS343`
[ Info: ode_configuration: 822.960 ms (921421 allocations: 60.39 MiB)
[ Info: get_callbacks: 115.899 ms (94339 allocations: 6.09 MiB)
[ Info: initializing diagnostics: 1.108 s (2543375 allocations: 130.26 MiB)
[ Info: Prepared diagnostic callbacks: 208.033 ms (314419 allocations: 16.13 MiB)
[ Info: Prepared SciMLBase.CallbackSet callbacks: 7.296 ms (1624 allocations: 110.04 KiB)
[ Info: n_steps_per_cycle_per_cb (non diagnostics): [864, 1]
[ Info: n_steps_per_cycle_per_cb_diagnostic: Any[]
[ Info: n_steps_per_cycle (non diagnostics): 864
[ Info: Define ode function: 5.046 s (415532 allocations: 28.68 MiB)
[ Info: dt_save_to_sol: Inf, length(saveat): 1
[ Info: Saving diagnostics to HDF5 file on day 0 second 0
[ Info: init integrator: 69.210 s (61083128 allocations: 3.72 GiB)
[ Info: Init diagnostics: 2.500 μs (0 allocations: 0 bytes)
┌ Info: Running
│   job_id = "target_gpu_implicit_moist_baroclinic_wave_mo"
│   output_dir = "output/target_gpu_implicit_moist_baroclinic_wave_mo"
└   tspan = (0.0f0, 1.296f6)
[ Info: Saving diagnostics to HDF5 file on day 1 second 0
[ Info: Saving diagnostics to HDF5 file on day 2 second 0
[ Info: Saving diagnostics to HDF5 file on day 3 second 0
...

simonbyrne commented 1 year ago

Dierckx.Spline1D isn't going to work on the GPU. We can either figure out some other mechanism to construct topography, or we initialize it all the CPU (#2241)

akshaysridhar commented 1 year ago

Dierckx.Spline1D isn't going to work on the GPU. We can either figure out some other mechanism to construct topography, or we initialize it all the CPU (#2241)

Thanks @simonbyrne This specific issue appears to be in the radiation_cache here:

            # interpolate the ozone concentrations to our initial pressures
            pressure2ozone = Spline1D(
                input_center_pressure,
                input_center_volume_mixing_ratio_o3,
            )

(Clearsky radiation parameterisation) This is in the ClimaAtmos interface as opposed to the radiation source code. We expect to hit similar issues when enabling topography. cc @sriharshakandala

simonbyrne commented 1 year ago

See #2257

akshaysridhar commented 1 year ago

@sriharshakandala With the CPU init updates, we can show that SF is compatible with the ClimaAtmos interfaces as in as/up-sf-fluxes > commit 7325e5b0 (running on single A100s). I think we can close this (since the other interpolation or GPU initialization issues are tracked elsewhere).

CliMA / ClimaAtmos.jl

Testing GPU compatible SF in ClimaAtmos #2235