Add support for `MtlArrays`?

[navidcy]

https://juliagpu.org/post/2022-06-24-metal/

[vchuravy]

One would first need to add a KernelAbstractions backend. Contributions welcome.

[jagoosw]

KernelAbstractions now has Metal backends https://github.com/JuliaGPU/KernelAbstractions.jl !

[jagoosw]

I started trying to add this support here: https://github.com/CliMA/Oceananigans.jl/tree/jsw/metal

But have hit the block of there being no FFT libraries for Metal yet (according to https://discourse.julialang.org/t/metal-jl-does-not-speed-up-fft/95528/3).

Also Metal only supports single precision.

[glwagner]

Maybe try the hydrostatic model or shallow water model ?

[jagoosw]

For hydrostatic models I get errors about trying to pass double precision numbers to a kernel

[glwagner]

@simone-silvestri has used the hydrostatic model in single precision. We should make it easier if it is tricky...

[jagoosw]

I think the double precision numbers are coming from Nx/Ny/Nz being turned into Int64 (even if I manually set them to be Int32) when the grid is constructed. I can't work out exactly where the promotion is happening though. This happens because Nx etc. are ::Int type in the grid, so get promoted when the grid is made, if I change them to be like the floats and be IT instead then the promotion doesn't happen. When I try to make the model I get loads of other things converted to Int64 though so still doesn't work.

[glwagner]

Why is that only an issue for hydrostatic? Are you just hitting an error sooner with nonhydrostatic?

[jagoosw]

Yeah I think so

[glwagner]

I think you can change the meaning of Int when you compile julia.

There might be an issue if your system is 64 bit. Then even if Nx::Int32, when you write

Nx + 1

the expression would be up converted to Int64 because 1 is Int64.

[glwagner]

Another possibility is that we refactor the code to generalize integer types. Which is not actually a horrible idea. There's some notion that we may also benefit from index types that are Int32. Adding an integer type for Nx would allow us to also automatically convert integers to Int32 (eg we write something like i = inttype(grid)).

[glwagner]

For hydrostatic models I get errors about trying to pass double precision numbers to a kernel

The Metal examples don't say anything about this btw. Can you please paste the error that you find into this issue?

[jagoosw]

I get this with the default free surface model (which I think can be solved fairly easily):

model = HydrostaticFreeSurfaceModel(; grid)
ERROR: Metal does not support Float64 values, try using Float32 instead
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] check_eltype(T::Type)
    @ Metal ~/.julia/packages/Metal/lnkVP/src/array.jl:32
  [3] Metal.MtlArray{Float64, 3, Metal.MTL.MTLResourceStorageModePrivate}(#unused#::UndefInitializer, dims::Tuple{Int64, Int64, Int64})
    @ Metal ~/.julia/packages/Metal/lnkVP/src/array.jl:50
  [4] (Metal.MtlArray{Float64, 3})(#unused#::UndefInitializer, dims::Tuple{Int64, Int64, Int64})
    @ Metal ~/.julia/packages/Metal/lnkVP/src/array.jl:98
  [5] MtlArray
    @ ~/.julia/packages/Metal/lnkVP/src/array.jl:157 [inlined]
  [6] Metal.MtlArray(A::Array{Float64, 3})
    @ Metal ~/.julia/packages/Metal/lnkVP/src/array.jl:173
  [7] arch_array(#unused#::Oceananigans.Architectures.MetalBackend, a::Array{Float64, 3})
    @ Oceananigans.Architectures ~/Documents/Projects/Oceananigans.jl/src/Architectures.jl:75
  [8] Oceananigans.Solvers.FFTBasedPoissonSolver(grid::RectilinearGrid{Float64, Periodic, Periodic, Flat, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, Oceananigans.Architectures.MetalBackend}, planner_flag::UInt32)
    @ Oceananigans.Solvers ~/Documents/Projects/Oceananigans.jl/src/Solvers/fft_based_poisson_solver.jl:61
  [9] Oceananigans.Solvers.FFTBasedPoissonSolver(grid::RectilinearGrid{Float64, Periodic, Periodic, Flat, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, Oceananigans.Architectures.MetalBackend})
    @ Oceananigans.Solvers ~/Documents/Projects/Oceananigans.jl/src/Solvers/fft_based_poisson_solver.jl:51
 [10] Oceananigans.Models.HydrostaticFreeSurfaceModels.FFTImplicitFreeSurfaceSolver(grid::RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Vector{Float32}}, OffsetArrays.OffsetVector{Float32, Vector{Float32}}, OffsetArrays.OffsetVector{Float32, Vector{Float32}}, Oceananigans.Architectures.MetalBackend}, settings::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, gravitational_acceleration::Float32)
    @ Oceananigans.Models.HydrostaticFreeSurfaceModels ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/fft_based_implicit_free_surface_solver.jl:67
 [11] build_implicit_step_solver
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/fft_based_implicit_free_surface_solver.jl:73 [inlined]
 [12] build_implicit_step_solver(#unused#::Val{:Default}, grid::RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Vector{Float32}}, OffsetArrays.OffsetVector{Float32, Vector{Float32}}, OffsetArrays.OffsetVector{Float32, Vector{Float32}}, Oceananigans.Architectures.MetalBackend}, settings::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, gravitational_acceleration::Float32)
    @ Oceananigans.Models.HydrostaticFreeSurfaceModels ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/implicit_free_surface.jl:111
 [13] FreeSurface(free_surface::ImplicitFreeSurface{Nothing, Float64, Nothing, Nothing, Symbol, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}}, velocities::NamedTuple{(:u, :v, :w), Tuple{Field{Face, Center, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Vector{Float32}}, OffsetArrays.OffsetVector{Float32, Vector{Float32}}, OffsetArrays.OffsetVector{Float32, Vector{Float32}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Face, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Vector{Float32}}, OffsetArrays.OffsetVector{Float32, Vector{Float32}}, OffsetArrays.OffsetVector{Float32, Vector{Float32}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Face, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Vector{Float32}}, OffsetArrays.OffsetVector{Float32, Vector{Float32}}, OffsetArrays.OffsetVector{Float32, Vector{Float32}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}}, grid::RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Vector{Float32}}, OffsetArrays.OffsetVector{Float32, Vector{Float32}}, OffsetArrays.OffsetVector{Float32, Vector{Float32}}, Oceananigans.Architectures.MetalBackend})
    @ Oceananigans.Models.HydrostaticFreeSurfaceModels ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/implicit_free_surface.jl:95
 [14] HydrostaticFreeSurfaceModel(; grid::RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Vector{Float32}}, OffsetArrays.OffsetVector{Float32, Vector{Float32}}, OffsetArrays.OffsetVector{Float32, Vector{Float32}}, Oceananigans.Architectures.MetalBackend}, clock::Clock{Float32}, momentum_advection::Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}, tracer_advection::Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}, buoyancy::SeawaterBuoyancy{Float32, LinearEquationOfState{Float32}, Nothing, Nothing}, coriolis::Nothing, free_surface::ImplicitFreeSurface{Nothing, Float64, Nothing, Nothing, Symbol, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}}, forcing::NamedTuple{(), Tuple{}}, closure::Nothing, boundary_conditions::NamedTuple{(), Tuple{}}, tracers::Tuple{Symbol, Symbol}, particles::Nothing, biogeochemistry::Nothing, velocities::Nothing, pressure::Nothing, diffusivity_fields::Nothing, auxiliary_fields::NamedTuple{(), Tuple{}})
    @ Oceananigans.Models.HydrostaticFreeSurfaceModels ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/hydrostatic_free_surface_model.jl:169
 [15] top-level scope
    @ REPL[5]:1
 [16] top-level scope
    @ ~/.julia/packages/Metal/lnkVP/src/initialization.jl:57

And with ExplicitFreeSurface I get:

Reason: unsupported use of double value
Stacktrace:
  [1] Float64
    @ ./float.jl:261
  [2] convert
    @ ./number.jl:7
  [3] _promote
    @ ./promotion.jl:358
  [4] promote
    @ ./promotion.jl:381
  [5] *
    @ ./promotion.jl:411
  [6] *
    @ ./operators.jl:578
  [7] advective_momentum_flux_Wu
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/centered_advective_fluxes.jl:17
  [8] _advective_momentum_flux_Wu
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/momentum_advection_operators.jl:10
  [9] δzᵃᵃᶜ
    @ ~/Documents/Projects/Oceananigans.jl/src/Operators/difference_operators.jl:26
 [10] div_𝐯u
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/momentum_advection_operators.jl:55
 [11] U_dot_∇u
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/vector_invariant_advection.jl:282
 [12] hydrostatic_free_surface_u_velocity_tendency
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/hydrostatic_free_surface_tendency_kernel_functions.jl:47
 [13] macro expansion
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/compute_hydrostatic_free_surface_tendencies.jl:236
 [14] gpu_compute_hydrostatic_free_surface_Gu!
    @ ~/.julia/packages/KernelAbstractions/cWlFz/src/macros.jl:90
 [15] gpu_compute_hydrostatic_free_surface_Gu!
    @ ./none:0
Reason: unsupported use of double value
Stacktrace:
  [1] *
    @ ./float.jl:410
  [2] *
    @ ./promotion.jl:411
  [3] *
    @ ./operators.jl:578
  [4] advective_momentum_flux_Wu
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/centered_advective_fluxes.jl:17
  [5] _advective_momentum_flux_Wu
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/momentum_advection_operators.jl:10
  [6] δzᵃᵃᶜ
    @ ~/Documents/Projects/Oceananigans.jl/src/Operators/difference_operators.jl:26
  [7] div_𝐯u
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/momentum_advection_operators.jl:55
  [8] U_dot_∇u
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/vector_invariant_advection.jl:282
  [9] hydrostatic_free_surface_u_velocity_tendency
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/hydrostatic_free_surface_tendency_kernel_functions.jl:47
 [10] macro expansion
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/compute_hydrostatic_free_surface_tendencies.jl:236
 [11] gpu_compute_hydrostatic_free_surface_Gu!
    @ ~/.julia/packages/KernelAbstractions/cWlFz/src/macros.jl:90
 [12] gpu_compute_hydrostatic_free_surface_Gu!
    @ ./none:0
Reason: unsupported use of double value
Stacktrace:
  [1] *
    @ ./float.jl:410
  [2] *
    @ ./operators.jl:578
  [3] advective_momentum_flux_Wu
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/centered_advective_fluxes.jl:17
  [4] _advective_momentum_flux_Wu
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/momentum_advection_operators.jl:10
  [5] δzᵃᵃᶜ
    @ ~/Documents/Projects/Oceananigans.jl/src/Operators/difference_operators.jl:26
  [6] div_𝐯u
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/momentum_advection_operators.jl:55
  [7] U_dot_∇u
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/vector_invariant_advection.jl:282
  [8] hydrostatic_free_surface_u_velocity_tendency
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/hydrostatic_free_surface_tendency_kernel_functions.jl:47
  [9] macro expansion
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/compute_hydrostatic_free_surface_tendencies.jl:236
 [10] gpu_compute_hydrostatic_free_surface_Gu!
    @ ~/.julia/packages/KernelAbstractions/cWlFz/src/macros.jl:90
 [11] gpu_compute_hydrostatic_free_surface_Gu!
    @ ./none:0
Reason: unsupported use of double value
Stacktrace:
  [1] Float64
    @ ./float.jl:261
  [2] convert
    @ ./number.jl:7
  [3] _promote
    @ ./promotion.jl:358
  [4] promote
    @ ./promotion.jl:381
  [5] *
    @ ./promotion.jl:411
  [6] inner_symmetric_interpolate_xᶠᵃᵃ
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/centered_reconstruction.jl:107
  [7] symmetric_interpolate_xᶠᵃᵃ
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/reconstruction_coefficients.jl:3
  [8] _symmetric_interpolate_xᶠᵃᵃ
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/topologically_conditional_interpolation.jl:60
  [9] advective_momentum_flux_Wu
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/centered_advective_fluxes.jl:17
 [10] _advective_momentum_flux_Wu
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/momentum_advection_operators.jl:10
 [11] δzᵃᵃᶜ
    @ ~/Documents/Projects/Oceananigans.jl/src/Operators/difference_operators.jl:26
 [12] div_𝐯u
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/momentum_advection_operators.jl:55
 [13] U_dot_∇u
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/vector_invariant_advection.jl:282
 [14] hydrostatic_free_surface_u_velocity_tendency
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/hydrostatic_free_surface_tendency_kernel_functions.jl:47
 [15] macro expansion
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/compute_hydrostatic_free_surface_tendencies.jl:236
 [16] gpu_compute_hydrostatic_free_surface_Gu!
    @ ~/.julia/packages/KernelAbstractions/cWlFz/src/macros.jl:90
 [17] gpu_compute_hydrostatic_free_surface_Gu!
    @ ./none:0
Reason: unsupported use of double value
Stacktrace:
  [1] *
    @ ./float.jl:410
  [2] *
    @ ./promotion.jl:411
  [3] inner_symmetric_interpolate_xᶠᵃᵃ
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/centered_reconstruction.jl:107
  [4] symmetric_interpolate_xᶠᵃᵃ
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/reconstruction_coefficients.jl:3
  [5] _symmetric_interpolate_xᶠᵃᵃ
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/topologically_conditional_interpolation.jl:60
  [6] advective_momentum_flux_Wu
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/centered_advective_fluxes.jl:17
  [7] _advective_momentum_flux_Wu
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/momentum_advection_operators.jl:10
  [8] δzᵃᵃᶜ
    @ ~/Documents/Projects/Oceananigans.jl/src/Operators/difference_operators.jl:26
  [9] div_𝐯u
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/momentum_advection_operators.jl:55
 [10] U_dot_∇u
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/vector_invariant_advection.jl:282
 [11] hydrostatic_free_surface_u_velocity_tendency
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/hydrostatic_free_surface_tendency_kernel_functions.jl:47
 [12] macro expansion
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/compute_hydrostatic_free_surface_tendencies.jl:236
 [13] gpu_compute_hydrostatic_free_surface_Gu!
    @ ~/.julia/packages/KernelAbstractions/cWlFz/src/macros.jl:90
 [14] gpu_compute_hydrostatic_free_surface_Gu!
    @ ./none:0
Reason: unsupported use of double value
Stacktrace:
  [1] Float64
    @ ./float.jl:261
  [2] convert
    @ ./number.jl:7
  [3] _promote
    @ ./promotion.jl:358
  [4] promote
    @ ./promotion.jl:381
  [5] *
    @ ./promotion.jl:411
  [6] inner_symmetric_interpolate_xᶠᵃᵃ
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/centered_reconstruction.jl:107
  [7] symmetric_interpolate_xᶠᵃᵃ
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/reconstruction_coefficients.jl:3
  [8] _symmetric_interpolate_xᶠᵃᵃ
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/topologically_conditional_interpolation.jl:60
  [9] advective_momentum_flux_Wu
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/centered_advective_fluxes.jl:17
 [10] _advective_momentum_flux_Wu
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/momentum_advection_operators.jl:10
 [11] δzᵃᵃᶜ
    @ ~/Documents/Projects/Oceananigans.jl/src/Operators/difference_operators.jl:26
 [12] div_𝐯u
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/momentum_advection_operators.jl:55
 [13] U_dot_∇u
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/vector_invariant_advection.jl:282
 [14] hydrostatic_free_surface_u_velocity_tendency
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/hydrostatic_free_surface_tendency_kernel_functions.jl:47
 [15] macro expansion
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/compute_hydrostatic_free_surface_tendencies.jl:236
 [16] gpu_compute_hydrostatic_free_surface_Gu!
    @ ~/.julia/packages/KernelAbstractions/cWlFz/src/macros.jl:90
 [17] gpu_compute_hydrostatic_free_surface_Gu!
    @ ./none:0
Reason: unsupported use of double value
Stacktrace:
  [1] *
    @ ./float.jl:410
  [2] *
    @ ./promotion.jl:411
  [3] inner_symmetric_interpolate_xᶠᵃᵃ
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/centered_reconstruction.jl:107
  [4] symmetric_interpolate_xᶠᵃᵃ
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/reconstruction_coefficients.jl:3
  [5] _symmetric_interpolate_xᶠᵃᵃ
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/topologically_conditional_interpolation.jl:60
  [6] advective_momentum_flux_Wu
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/centered_advective_fluxes.jl:17
  [7] _advective_momentum_flux_Wu
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/momentum_advection_operators.jl:10
  [8] δzᵃᵃᶜ
    @ ~/Documents/Projects/Oceananigans.jl/src/Operators/difference_operators.jl:26
  [9] div_𝐯u
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/momentum_advection_operators.jl:55
 [10] U_dot_∇u
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/vector_invariant_advection.jl:282
 [11] hydrostatic_free_surface_u_velocity_tendency
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/hydrostatic_free_surface_tendency_kernel_functions.jl:47
 [12] macro expansion
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/compute_hydrostatic_free_surface_tendencies.jl:236
 [13] gpu_compute_hydrostatic_free_surface_Gu!
    @ ~/.julia/packages/KernelAbstractions/cWlFz/src/macros.jl:90
 [14] gpu_compute_hydrostatic_free_surface_Gu!
    @ ./none:0
Reason: unsupported use of double value
Stacktrace:
  [1] +
    @ ./float.jl:408
  [2] inner_symmetric_interpolate_xᶠᵃᵃ
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/centered_reconstruction.jl:107
  [3] symmetric_interpolate_xᶠᵃᵃ
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/reconstruction_coefficients.jl:3
  [4] _symmetric_interpolate_xᶠᵃᵃ
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/topologically_conditional_interpolation.jl:60
  [5] advective_momentum_flux_Wu
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/centered_advective_fluxes.jl:17
  [6] _advective_momentum_flux_Wu
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/momentum_advection_operators.jl:10
  [7] δzᵃᵃᶜ
    @ ~/Documents/Projects/Oceananigans.jl/src/Operators/difference_operators.jl:26
  [8] div_𝐯u
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/momentum_advection_operators.jl:55
  [9] U_dot_∇u
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/vector_invariant_advection.jl:282
 [10] hydrostatic_free_surface_u_velocity_tendency
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/hydrostatic_free_surface_tendency_kernel_functions.jl:47
 [11] macro expansion
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/compute_hydrostatic_free_surface_tendencies.jl:236
 [12] gpu_compute_hydrostatic_free_surface_Gu!
    @ ~/.julia/packages/KernelAbstractions/cWlFz/src/macros.jl:90
 [13] gpu_compute_hydrostatic_free_surface_Gu!
    @ ./none:0
Reason: unsupported use of double value
Stacktrace:
  [1] Float64
    @ ./float.jl:261
  [2] convert
    @ ./number.jl:7
  [3] _promote
    @ ./promotion.jl:358
  [4] promote
    @ ./promotion.jl:381
  [5] *
    @ ./promotion.jl:411
  [6] inner_symmetric_interpolate_zᵃᵃᶠ
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/centered_reconstruction.jl:113
  [7] symmetric_interpolate_zᵃᵃᶠ
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/reconstruction_coefficients.jl:5
  [8] _symmetric_interpolate_zᵃᵃᶠ
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/topologically_conditional_interpolation.jl:60
  [9] advective_momentum_flux_Wu
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/centered_advective_fluxes.jl:17
 [10] _advective_momentum_flux_Wu
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/momentum_advection_operators.jl:10
 [11] δzᵃᵃᶜ
    @ ~/Documents/Projects/Oceananigans.jl/src/Operators/difference_operators.jl:26
 [12] div_𝐯u
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/momentum_advection_operators.jl:55
 [13] U_dot_∇u
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/vector_invariant_advection.jl:282
 [14] hydrostatic_free_surface_u_velocity_tendency
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/hydrostatic_free_surface_tendency_kernel_functions.jl:47
 [15] macro expansion
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/compute_hydrostatic_free_surface_tendencies.jl:236
 [16] gpu_compute_hydrostatic_free_surface_Gu!
    @ ~/.julia/packages/KernelAbstractions/cWlFz/src/macros.jl:90
 [17] gpu_compute_hydrostatic_free_surface_Gu!
    @ ./none:0
Reason: unsupported use of double value
Stacktrace:
  [1] *
    @ ./float.jl:410
  [2] *
    @ ./promotion.jl:411
  [3] inner_symmetric_interpolate_zᵃᵃᶠ
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/centered_reconstruction.jl:113
  [4] symmetric_interpolate_zᵃᵃᶠ
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/reconstruction_coefficients.jl:5
  [5] _symmetric_interpolate_zᵃᵃᶠ
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/topologically_conditional_interpolation.jl:60
  [6] advective_momentum_flux_Wu
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/centered_advective_fluxes.jl:17
  [7] _advective_momentum_flux_Wu
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/momentum_advection_operators.jl:10
  [8] δzᵃᵃᶜ
    @ ~/Documents/Projects/Oceananigans.jl/src/Operators/difference_operators.jl:26
  [9] div_𝐯u
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/momentum_advection_operators.jl:55
 [10] U_dot_∇u
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/vector_invariant_advection.jl:282
 [11] hydrostatic_free_surface_u_velocity_tendency
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/hydrostatic_free_surface_tendency_kernel_functions.jl:47
 [12] macro expansion
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/compute_hydrostatic_free_surface_tendencies.jl:236
 [13] gpu_compute_hydrostatic_free_surface_Gu!
    @ ~/.julia/packages/KernelAbstractions/cWlFz/src/macros.jl:90
 [14] gpu_compute_hydrostatic_free_surface_Gu!
    @ ./none:0
Reason: unsupported use of double value
Stacktrace:
  [1] +
    @ ./float.jl:408
  [2] inner_symmetric_interpolate_zᵃᵃᶠ
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/centered_reconstruction.jl:113
  [3] symmetric_interpolate_zᵃᵃᶠ
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/reconstruction_coefficients.jl:5
  [4] _symmetric_interpolate_zᵃᵃᶠ
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/topologically_conditional_interpolation.jl:60
  [5] advective_momentum_flux_Wu
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/centered_advective_fluxes.jl:17
  [6] _advective_momentum_flux_Wu
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/momentum_advection_operators.jl:10
  [7] δzᵃᵃᶜ
    @ ~/Documents/Projects/Oceananigans.jl/src/Operators/difference_operators.jl:26
  [8] div_𝐯u
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/momentum_advection_operators.jl:55
  [9] U_dot_∇u
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/vector_invariant_advection.jl:282
 [10] hydrostatic_free_surface_u_velocity_tendency
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/hydrostatic_free_surface_tendency_kernel_functions.jl:47
 [11] macro expansion
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/compute_hydrostatic_free_surface_tendencies.jl:236
 [12] gpu_compute_hydrostatic_free_surface_Gu!
    @ ~/.julia/packages/KernelAbstractions/cWlFz/src/macros.jl:90
 [13] gpu_compute_hydrostatic_free_surface_Gu!
    @ ./none:0
Reason: unsupported use of double value
Stacktrace:
  [1] *
    @ ./float.jl:410
  [2] *
    @ ./promotion.jl:411
  [3] *
    @ ./operators.jl:578
  [4] advective_momentum_flux_Wu
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/centered_advective_fluxes.jl:17
  [5] _advective_momentum_flux_Wu
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/momentum_advection_operators.jl:10
  [6] δzᵃᵃᶜ
    @ ~/Documents/Projects/Oceananigans.jl/src/Operators/difference_operators.jl:26
  [7] div_𝐯u
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/momentum_advection_operators.jl:55
  [8] U_dot_∇u
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/vector_invariant_advection.jl:282
  [9] hydrostatic_free_surface_u_velocity_tendency
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/hydrostatic_free_surface_tendency_kernel_functions.jl:47
 [10] macro expansion
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/compute_hydrostatic_free_surface_tendencies.jl:236
 [11] gpu_compute_hydrostatic_free_surface_Gu!
    @ ~/.julia/packages/KernelAbstractions/cWlFz/src/macros.jl:90
 [12] gpu_compute_hydrostatic_free_surface_Gu!
    @ ./none:0
Reason: unsupported use of double value
Stacktrace:
  [1] *
    @ ./float.jl:410
  [2] *
    @ ./operators.jl:578
  [3] advective_momentum_flux_Wu
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/centered_advective_fluxes.jl:17
  [4] _advective_momentum_flux_Wu
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/momentum_advection_operators.jl:10
  [5] δzᵃᵃᶜ
    @ ~/Documents/Projects/Oceananigans.jl/src/Operators/difference_operators.jl:26
  [6] div_𝐯u
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/momentum_advection_operators.jl:55
  [7] U_dot_∇u
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/vector_invariant_advection.jl:282
  [8] hydrostatic_free_surface_u_velocity_tendency
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/hydrostatic_free_surface_tendency_kernel_functions.jl:47
  [9] macro expansion
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/compute_hydrostatic_free_surface_tendencies.jl:236
 [10] gpu_compute_hydrostatic_free_surface_Gu!
    @ ~/.julia/packages/KernelAbstractions/cWlFz/src/macros.jl:90
 [11] gpu_compute_hydrostatic_free_surface_Gu!
    @ ./none:0
Reason: unsupported use of double value
Stacktrace:
 [1] -
   @ ./float.jl:409
 [2] δzᵃᵃᶜ
   @ ~/Documents/Projects/Oceananigans.jl/src/Operators/difference_operators.jl:26
 [3] div_𝐯u
   @ ~/Documents/Projects/Oceananigans.jl/src/Advection/momentum_advection_operators.jl:55
 [4] U_dot_∇u
   @ ~/Documents/Projects/Oceananigans.jl/src/Advection/vector_invariant_advection.jl:282
 [5] hydrostatic_free_surface_u_velocity_tendency
   @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/hydrostatic_free_surface_tendency_kernel_functions.jl:47
 [6] macro expansion
   @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/compute_hydrostatic_free_surface_tendencies.jl:236
 [7] gpu_compute_hydrostatic_free_surface_Gu!
   @ ~/.julia/packages/KernelAbstractions/cWlFz/src/macros.jl:90
 [8] gpu_compute_hydrostatic_free_surface_Gu!
   @ ./none:0
Reason: unsupported use of double value
Stacktrace:
 [1] +
   @ ./float.jl:408
 [2] +
   @ ./operators.jl:578
 [3] div_𝐯u
   @ ~/Documents/Projects/Oceananigans.jl/src/Advection/momentum_advection_operators.jl:55
 [4] U_dot_∇u
   @ ~/Documents/Projects/Oceananigans.jl/src/Advection/vector_invariant_advection.jl:282
 [5] hydrostatic_free_surface_u_velocity_tendency
   @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/hydrostatic_free_surface_tendency_kernel_functions.jl:47
 [6] macro expansion
   @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/compute_hydrostatic_free_surface_tendencies.jl:236
 [7] gpu_compute_hydrostatic_free_surface_Gu!
   @ ~/.julia/packages/KernelAbstractions/cWlFz/src/macros.jl:90
 [8] gpu_compute_hydrostatic_free_surface_Gu!
   @ ./none:0
Reason: unsupported use of double value
Stacktrace:
 [1] +
   @ ./float.jl:408
 [2] +
   @ ./operators.jl:578
 [3] div_𝐯u
   @ ~/Documents/Projects/Oceananigans.jl/src/Advection/momentum_advection_operators.jl:55
 [4] U_dot_∇u
   @ ~/Documents/Projects/Oceananigans.jl/src/Advection/vector_invariant_advection.jl:282
 [5] hydrostatic_free_surface_u_velocity_tendency
   @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/hydrostatic_free_surface_tendency_kernel_functions.jl:47
 [6] macro expansion
   @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/compute_hydrostatic_free_surface_tendencies.jl:236
 [7] gpu_compute_hydrostatic_free_surface_Gu!
   @ ~/.julia/packages/KernelAbstractions/cWlFz/src/macros.jl:90
 [8] gpu_compute_hydrostatic_free_surface_Gu!
   @ ./none:0
Reason: unsupported use of double value
Stacktrace:
  [1] Float64
    @ ./float.jl:261
  [2] convert
    @ ./number.jl:7
  [3] _promote
    @ ./promotion.jl:358
  [4] promote
    @ ./promotion.jl:381
  [5] *
    @ ./promotion.jl:411
  [6] div_𝐯u
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/momentum_advection_operators.jl:55
  [7] U_dot_∇u
    @ ~/Documents/Projects/Oceananigans.jl/src/Advection/vector_invariant_advection.jl:282
  [8] hydrostatic_free_surface_u_velocity_tendency
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/hydrostatic_free_surface_tendency_kernel_functions.jl:47
  [9] macro expansion
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/compute_hydrostatic_free_surface_tendencies.jl:236
 [10] gpu_compute_hydrostatic_free_surface_Gu!
    @ ~/.julia/packages/KernelAbstractions/cWlFz/src/macros.jl:90
 [11] gpu_compute_hydrostatic_free_surface_Gu!
    @ ./none:0
Reason: unsupported use of double value
Stacktrace:
 [1] -
   @ ./float.jl:406
 [2] hydrostatic_free_surface_u_velocity_tendency
   @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/hydrostatic_free_surface_tendency_kernel_functions.jl:47
 [3] macro expansion
   @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/compute_hydrostatic_free_surface_tendencies.jl:236
 [4] gpu_compute_hydrostatic_free_surface_Gu!
   @ ~/.julia/packages/KernelAbstractions/cWlFz/src/macros.jl:90
 [5] gpu_compute_hydrostatic_free_surface_Gu!
   @ ./none:0
Reason: unsupported use of double value
Stacktrace:
 [1] -
   @ ./float.jl:406
 [2] hydrostatic_free_surface_u_velocity_tendency
   @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/hydrostatic_free_surface_tendency_kernel_functions.jl:47
 [3] macro expansion
   @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/compute_hydrostatic_free_surface_tendencies.jl:236
 [4] gpu_compute_hydrostatic_free_surface_Gu!
   @ ~/.julia/packages/KernelAbstractions/cWlFz/src/macros.jl:90
 [5] gpu_compute_hydrostatic_free_surface_Gu!
   @ ./none:0
Reason: unsupported use of double value
Stacktrace:
 [1] Float64
   @ ./float.jl:261
 [2] convert
   @ ./number.jl:7
 [3] _promote
   @ ./promotion.jl:358
 [4] promote
   @ ./promotion.jl:381
 [5] -
   @ ./promotion.jl:412
 [6] hydrostatic_free_surface_u_velocity_tendency
   @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/hydrostatic_free_surface_tendency_kernel_functions.jl:47
 [7] macro expansion
   @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/compute_hydrostatic_free_surface_tendencies.jl:236
 [8] gpu_compute_hydrostatic_free_surface_Gu!
   @ ~/.julia/packages/KernelAbstractions/cWlFz/src/macros.jl:90
 [9] gpu_compute_hydrostatic_free_surface_Gu!
   @ ./none:0
Reason: unsupported use of double value
Stacktrace:
 [1] -
   @ ./float.jl:409
 [2] -
   @ ./promotion.jl:412
 [3] hydrostatic_free_surface_u_velocity_tendency
   @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/hydrostatic_free_surface_tendency_kernel_functions.jl:47
 [4] macro expansion
   @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/compute_hydrostatic_free_surface_tendencies.jl:236
 [5] gpu_compute_hydrostatic_free_surface_Gu!
   @ ~/.julia/packages/KernelAbstractions/cWlFz/src/macros.jl:90
 [6] gpu_compute_hydrostatic_free_surface_Gu!
   @ ./none:0
Reason: unsupported use of double value
Stacktrace:
 [1] Float64
   @ ./float.jl:261
 [2] convert
   @ ./number.jl:7
 [3] _promote
   @ ./promotion.jl:358
 [4] promote
   @ ./promotion.jl:381
 [5] -
   @ ./promotion.jl:412
 [6] hydrostatic_free_surface_u_velocity_tendency
   @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/hydrostatic_free_surface_tendency_kernel_functions.jl:47
 [7] macro expansion
   @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/compute_hydrostatic_free_surface_tendencies.jl:236
 [8] gpu_compute_hydrostatic_free_surface_Gu!
   @ ~/.julia/packages/KernelAbstractions/cWlFz/src/macros.jl:90
 [9] gpu_compute_hydrostatic_free_surface_Gu!
   @ ./none:0
Reason: unsupported use of double value
Stacktrace:
 [1] -
   @ ./float.jl:409
 [2] -
   @ ./promotion.jl:412
 [3] hydrostatic_free_surface_u_velocity_tendency
   @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/hydrostatic_free_surface_tendency_kernel_functions.jl:47
 [4] macro expansion
   @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/compute_hydrostatic_free_surface_tendencies.jl:236
 [5] gpu_compute_hydrostatic_free_surface_Gu!
   @ ~/.julia/packages/KernelAbstractions/cWlFz/src/macros.jl:90
 [6] gpu_compute_hydrostatic_free_surface_Gu!
   @ ./none:0
Reason: unsupported use of double value
Stacktrace:
 [1] +
   @ ./float.jl:408
 [2] +
   @ ./promotion.jl:410
 [3] hydrostatic_free_surface_u_velocity_tendency
   @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/hydrostatic_free_surface_tendency_kernel_functions.jl:47
 [4] macro expansion
   @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/compute_hydrostatic_free_surface_tendencies.jl:236
 [5] gpu_compute_hydrostatic_free_surface_Gu!
   @ ~/.julia/packages/KernelAbstractions/cWlFz/src/macros.jl:90
 [6] gpu_compute_hydrostatic_free_surface_Gu!
   @ ./none:0
Reason: unsupported use of double value
Stacktrace:
 [1] Float32
   @ ./float.jl:258
 [2] convert
   @ ./number.jl:7
 [3] setindex!
   @ ~/.julia/packages/Metal/lnkVP/src/device/array.jl:105
 [4] setindex!
   @ ~/.julia/packages/Metal/lnkVP/src/device/array.jl:118
 [5] setindex!
   @ ~/.julia/packages/OffsetArrays/0MOrf/src/OffsetArrays.jl:443
 [6] macro expansion
   @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/compute_hydrostatic_free_surface_tendencies.jl:236
 [7] gpu_compute_hydrostatic_free_surface_Gu!
   @ ~/.julia/packages/KernelAbstractions/cWlFz/src/macros.jl:90
 [8] gpu_compute_hydrostatic_free_surface_Gu!
   @ ./none:0
Hint: catch this exception as `err` and call `code_typed(err; interactive = true)` to introspect the erronous code with Cthulhu.jl
Stacktrace:
  [1] check_ir(job::GPUCompiler.CompilerJob{GPUCompiler.MetalCompilerTarget, Metal.MetalCompilerParams}, args::LLVM.Module)
    @ GPUCompiler ~/.julia/packages/GPUCompiler/Cp7sE/src/validation.jl:147
  [2] macro expansion
    @ ~/.julia/packages/GPUCompiler/Cp7sE/src/driver.jl:440 [inlined]
  [3] macro expansion
    @ ~/.julia/packages/TimerOutputs/RsWnF/src/TimerOutput.jl:253 [inlined]
  [4] macro expansion
    @ ~/.julia/packages/GPUCompiler/Cp7sE/src/driver.jl:439 [inlined]
  [5] emit_llvm(job::GPUCompiler.CompilerJob; libraries::Bool, toplevel::Bool, optimize::Bool, cleanup::Bool, only_entry::Bool, validate::Bool)
    @ GPUCompiler ~/.julia/packages/GPUCompiler/Cp7sE/src/utils.jl:89
  [6] emit_llvm
    @ ~/.julia/packages/GPUCompiler/Cp7sE/src/utils.jl:83 [inlined]
  [7] codegen(output::Symbol, job::GPUCompiler.CompilerJob; libraries::Bool, toplevel::Bool, optimize::Bool, cleanup::Bool, strip::Bool, validate::Bool, only_entry::Bool, parent_job::Nothing)
    @ GPUCompiler ~/.julia/packages/GPUCompiler/Cp7sE/src/driver.jl:129
  [8] codegen
    @ ~/.julia/packages/GPUCompiler/Cp7sE/src/driver.jl:110 [inlined]
  [9] compile(target::Symbol, job::GPUCompiler.CompilerJob; libraries::Bool, toplevel::Bool, optimize::Bool, cleanup::Bool, strip::Bool, validate::Bool, only_entry::Bool)
    @ GPUCompiler ~/.julia/packages/GPUCompiler/Cp7sE/src/driver.jl:106
 [10] compile
    @ ~/.julia/packages/GPUCompiler/Cp7sE/src/driver.jl:98 [inlined]
 [11] #45
    @ ~/.julia/packages/Metal/lnkVP/src/compiler/compilation.jl:57 [inlined]
 [12] JuliaContext(f::Metal.var"#45#46"{GPUCompiler.CompilerJob{GPUCompiler.MetalCompilerTarget, Metal.MetalCompilerParams}})
    @ GPUCompiler ~/.julia/packages/GPUCompiler/Cp7sE/src/driver.jl:47
 [13] compile(job::GPUCompiler.CompilerJob)
    @ Metal ~/.julia/packages/Metal/lnkVP/src/compiler/compilation.jl:56
 [14] actual_compilation(cache::Dict{Any, Any}, src::Core.MethodInstance, world::UInt64, cfg::GPUCompiler.CompilerConfig{GPUCompiler.MetalCompilerTarget, Metal.MetalCompilerParams}, compiler::typeof(Metal.compile), linker::typeof(Metal.link))
    @ GPUCompiler ~/.julia/packages/GPUCompiler/Cp7sE/src/execution.jl:125
 [15] cached_compilation(cache::Dict{Any, Any}, src::Core.MethodInstance, cfg::GPUCompiler.CompilerConfig{GPUCompiler.MetalCompilerTarget, Metal.MetalCompilerParams}, compiler::Function, linker::Function)
    @ GPUCompiler ~/.julia/packages/GPUCompiler/Cp7sE/src/execution.jl:103
 [16] macro expansion
    @ ~/.julia/packages/Metal/lnkVP/src/compiler/execution.jl:162 [inlined]
 [17] macro expansion
    @ ./lock.jl:267 [inlined]
 [18] mtlfunction(f::typeof(Oceananigans.Models.HydrostaticFreeSurfaceModels.gpu_compute_hydrostatic_free_surface_Gu!), tt::Type{Tuple{KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.StaticSize{(3, 3, 50)}, KernelAbstractions.NDIteration.DynamicCheck, Nothing, Nothing, KernelAbstractions.NDIteration.NDRange{3, KernelAbstractions.NDIteration.StaticSize{(1, 1, 50)}, KernelAbstractions.NDIteration.StaticSize{(16, 16, 1)}, Nothing, Nothing}}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlDeviceArray{Float32, 3, 1}}, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlDeviceVector{Float32, 1}}, OffsetArrays.OffsetVector{Float32, Metal.MtlDeviceVector{Float32, 1}}, OffsetArrays.OffsetVector{Float32, Metal.MtlDeviceVector{Float32, 1}}, Nothing}, Nothing, Tuple{Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, NamedTuple{(:u, :v, :w), Tuple{OffsetArrays.OffsetArray{Float32, 3, Metal.MtlDeviceArray{Float32, 3, 1}}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlDeviceArray{Float32, 3, 1}}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlDeviceArray{Float32, 3, 1}}}}, ExplicitFreeSurface{OffsetArrays.OffsetArray{Float32, 3, Metal.MtlDeviceArray{Float32, 3, 1}}, Float32}, NamedTuple{(:T, :S), Tuple{OffsetArrays.OffsetArray{Float32, 3, Metal.MtlDeviceArray{Float32, 3, 1}}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlDeviceArray{Float32, 3, 1}}}}, Buoyancy{SeawaterBuoyancy{Float32, LinearEquationOfState{Float32}, Nothing, Nothing}, Oceananigans.Grids.NegativeZDirection}, Nothing, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlDeviceArray{Float32, 3, 1}}, NamedTuple{(), Tuple{}}, NamedTuple{(:u, :v, :η, :T, :S), NTuple{5, typeof(Oceananigans.Forcings.zeroforcing)}}, NamedTuple{(:time, :iteration, :stage), Tuple{Float32, Int64, Int64}}}}}; name::Nothing, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Metal ~/.julia/packages/Metal/lnkVP/src/compiler/execution.jl:157
 [19] mtlfunction(f::typeof(Oceananigans.Models.HydrostaticFreeSurfaceModels.gpu_compute_hydrostatic_free_surface_Gu!), tt::Type{Tuple{KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.StaticSize{(3, 3, 50)}, KernelAbstractions.NDIteration.DynamicCheck, Nothing, Nothing, KernelAbstractions.NDIteration.NDRange{3, KernelAbstractions.NDIteration.StaticSize{(1, 1, 50)}, KernelAbstractions.NDIteration.StaticSize{(16, 16, 1)}, Nothing, Nothing}}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlDeviceArray{Float32, 3, 1}}, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlDeviceVector{Float32, 1}}, OffsetArrays.OffsetVector{Float32, Metal.MtlDeviceVector{Float32, 1}}, OffsetArrays.OffsetVector{Float32, Metal.MtlDeviceVector{Float32, 1}}, Nothing}, Nothing, Tuple{Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, NamedTuple{(:u, :v, :w), Tuple{OffsetArrays.OffsetArray{Float32, 3, Metal.MtlDeviceArray{Float32, 3, 1}}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlDeviceArray{Float32, 3, 1}}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlDeviceArray{Float32, 3, 1}}}}, ExplicitFreeSurface{OffsetArrays.OffsetArray{Float32, 3, Metal.MtlDeviceArray{Float32, 3, 1}}, Float32}, NamedTuple{(:T, :S), Tuple{OffsetArrays.OffsetArray{Float32, 3, Metal.MtlDeviceArray{Float32, 3, 1}}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlDeviceArray{Float32, 3, 1}}}}, Buoyancy{SeawaterBuoyancy{Float32, LinearEquationOfState{Float32}, Nothing, Nothing}, Oceananigans.Grids.NegativeZDirection}, Nothing, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlDeviceArray{Float32, 3, 1}}, NamedTuple{(), Tuple{}}, NamedTuple{(:u, :v, :η, :T, :S), NTuple{5, typeof(Oceananigans.Forcings.zeroforcing)}}, NamedTuple{(:time, :iteration, :stage), Tuple{Float32, Int64, Int64}}}}})
    @ Metal ~/.julia/packages/Metal/lnkVP/src/compiler/execution.jl:155
 [20] macro expansion
    @ ~/.julia/packages/Metal/lnkVP/src/compiler/execution.jl:77 [inlined]
 [21] (::KernelAbstractions.Kernel{Metal.MetalKernels.MetalBackend, KernelAbstractions.NDIteration.StaticSize{(16, 16)}, KernelAbstractions.NDIteration.StaticSize{(3, 3, 50)}, typeof(Oceananigans.Models.HydrostaticFreeSurfaceModels.gpu_compute_hydrostatic_free_surface_Gu!)})(::Field{Face, Center, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, ::Vararg{Any}; ndrange::Nothing, workgroupsize::Nothing)
    @ Metal.MetalKernels ~/.julia/packages/Metal/lnkVP/src/MetalKernels.jl:105
 [22] (::KernelAbstractions.Kernel{Metal.MetalKernels.MetalBackend, KernelAbstractions.NDIteration.StaticSize{(16, 16)}, KernelAbstractions.NDIteration.StaticSize{(3, 3, 50)}, typeof(Oceananigans.Models.HydrostaticFreeSurfaceModels.gpu_compute_hydrostatic_free_surface_Gu!)})(::Field{Face, Center, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, ::Vararg{Any})
    @ Metal.MetalKernels ~/.julia/packages/Metal/lnkVP/src/MetalKernels.jl:101
 [23] launch!(::Oceananigans.Architectures.MetalBackend, ::RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, ::Symbol, ::typeof(Oceananigans.Models.HydrostaticFreeSurfaceModels.compute_hydrostatic_free_surface_Gu!), ::Field{Face, Center, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, ::Vararg{Any}; include_right_boundaries::Bool, reduced_dimensions::Tuple{}, location::Nothing, only_active_cells::Nothing, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Oceananigans.Utils ~/Documents/Projects/Oceananigans.jl/src/Utils/kernel_launching.jl:119
 [24] launch!
    @ ~/Documents/Projects/Oceananigans.jl/src/Utils/kernel_launching.jl:90 [inlined]
 [25] #compute_hydrostatic_momentum_tendencies!#36
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/compute_hydrostatic_free_surface_tendencies.jl:194 [inlined]
 [26] compute_hydrostatic_momentum_tendencies!
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/compute_hydrostatic_free_surface_tendencies.jl:168 [inlined]
 [27] compute_hydrostatic_free_surface_tendency_contributions!(model::HydrostaticFreeSurfaceModel{Oceananigans.TimeSteppers.QuasiAdamsBashforth2TimeStepper{Float32, NamedTuple{(:u, :v, :η, :T, :S), Tuple{Field{Face, Center, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Face, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Face, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, UnitRange{Int64}}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}}, Nothing}, Nothing, Oceananigans.Architectures.MetalBackend, ExplicitFreeSurface{Field{Center, Center, Face, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, UnitRange{Int64}}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Float32}, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Float32, NamedTuple{(:momentum, :T, :S), Tuple{Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}, Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}, Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}}}, Buoyancy{SeawaterBuoyancy{Float32, LinearEquationOfState{Float32}, Nothing, Nothing}, Oceananigans.Grids.NegativeZDirection}, Nothing, NamedTuple{(:u, :v, :η, :T, :S), NTuple{5, typeof(Oceananigans.Forcings.zeroforcing)}}, Nothing, Nothing, NamedTuple{(:u, :v, :w), Tuple{Field{Face, Center, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Face, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Face, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}}, NamedTuple{(:T, :S), Tuple{Field{Center, Center, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}}, NamedTuple{(:pHY′,), Tuple{Field{Center, Center, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}}, Nothing, NamedTuple{(), Tuple{}}}, kernel_parameters::Tuple{Symbol}; only_active_cells::Nothing)
    @ Oceananigans.Models.HydrostaticFreeSurfaceModels ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/compute_hydrostatic_free_surface_tendencies.jl:95
 [28] compute_hydrostatic_free_surface_tendency_contributions!
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/compute_hydrostatic_free_surface_tendencies.jl:90 [inlined]
 [29] compute_tendencies!(model::HydrostaticFreeSurfaceModel{Oceananigans.TimeSteppers.QuasiAdamsBashforth2TimeStepper{Float32, NamedTuple{(:u, :v, :η, :T, :S), Tuple{Field{Face, Center, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Face, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Face, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, UnitRange{Int64}}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}}, Nothing}, Nothing, Oceananigans.Architectures.MetalBackend, ExplicitFreeSurface{Field{Center, Center, Face, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, UnitRange{Int64}}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Float32}, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Float32, NamedTuple{(:momentum, :T, :S), Tuple{Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}, Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}, Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}}}, Buoyancy{SeawaterBuoyancy{Float32, LinearEquationOfState{Float32}, Nothing, Nothing}, Oceananigans.Grids.NegativeZDirection}, Nothing, NamedTuple{(:u, :v, :η, :T, :S), NTuple{5, typeof(Oceananigans.Forcings.zeroforcing)}}, Nothing, Nothing, NamedTuple{(:u, :v, :w), Tuple{Field{Face, Center, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Face, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Face, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}}, NamedTuple{(:T, :S), Tuple{Field{Center, Center, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}}, NamedTuple{(:pHY′,), Tuple{Field{Center, Center, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}}, Nothing, NamedTuple{(), Tuple{}}}, callbacks::Vector{Any})
    @ Oceananigans.Models.HydrostaticFreeSurfaceModels ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/compute_hydrostatic_free_surface_tendencies.jl:31
 [30] #apply_regionally!#52
    @ ~/Documents/Projects/Oceananigans.jl/src/Utils/multi_region_transformation.jl:121 [inlined]
 [31] apply_regionally!
    @ ~/Documents/Projects/Oceananigans.jl/src/Utils/multi_region_transformation.jl:118 [inlined]
 [32] macro expansion
    @ ~/Documents/Projects/Oceananigans.jl/src/Utils/multi_region_transformation.jl:204 [inlined]
 [33] update_state!(model::HydrostaticFreeSurfaceModel{Oceananigans.TimeSteppers.QuasiAdamsBashforth2TimeStepper{Float32, NamedTuple{(:u, :v, :η, :T, :S), Tuple{Field{Face, Center, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Face, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Face, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, UnitRange{Int64}}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}}, Nothing}, Nothing, Oceananigans.Architectures.MetalBackend, ExplicitFreeSurface{Field{Center, Center, Face, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, UnitRange{Int64}}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Float32}, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Float32, NamedTuple{(:momentum, :T, :S), Tuple{Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}, Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}, Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}}}, Buoyancy{SeawaterBuoyancy{Float32, LinearEquationOfState{Float32}, Nothing, Nothing}, Oceananigans.Grids.NegativeZDirection}, Nothing, NamedTuple{(:u, :v, :η, :T, :S), NTuple{5, typeof(Oceananigans.Forcings.zeroforcing)}}, Nothing, Nothing, NamedTuple{(:u, :v, :w), Tuple{Field{Face, Center, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Face, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Face, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}}, NamedTuple{(:T, :S), Tuple{Field{Center, Center, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}}, NamedTuple{(:pHY′,), Tuple{Field{Center, Center, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}}, Nothing, NamedTuple{(), Tuple{}}}, grid::RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, callbacks::Vector{Any}; compute_tendencies::Bool)
    @ Oceananigans.Models.HydrostaticFreeSurfaceModels ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/update_hydrostatic_free_surface_model_state.jl:44
 [34] update_state!
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/update_hydrostatic_free_surface_model_state.jl:29 [inlined]
 [35] #update_state!#39
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/update_hydrostatic_free_surface_model_state.jl:26 [inlined]
 [36] update_state!
    @ ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/update_hydrostatic_free_surface_model_state.jl:26 [inlined]
 [37] update_state!(model::HydrostaticFreeSurfaceModel{Oceananigans.TimeSteppers.QuasiAdamsBashforth2TimeStepper{Float32, NamedTuple{(:u, :v, :η, :T, :S), Tuple{Field{Face, Center, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Face, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Face, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, UnitRange{Int64}}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}}, Nothing}, Nothing, Oceananigans.Architectures.MetalBackend, ExplicitFreeSurface{Field{Center, Center, Face, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, UnitRange{Int64}}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Float32}, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Float32, NamedTuple{(:momentum, :T, :S), Tuple{Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}, Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}, Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}}}, Buoyancy{SeawaterBuoyancy{Float32, LinearEquationOfState{Float32}, Nothing, Nothing}, Oceananigans.Grids.NegativeZDirection}, Nothing, NamedTuple{(:u, :v, :η, :T, :S), NTuple{5, typeof(Oceananigans.Forcings.zeroforcing)}}, Nothing, Nothing, NamedTuple{(:u, :v, :w), Tuple{Field{Face, Center, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Face, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Face, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}}, NamedTuple{(:T, :S), Tuple{Field{Center, Center, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}, Field{Center, Center, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}}, NamedTuple{(:pHY′,), Tuple{Field{Center, Center, Center, Nothing, RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, Tuple{Colon, Colon, Colon}, OffsetArrays.OffsetArray{Float32, 3, Metal.MtlArray{Float32, 3, Metal.MTL.MTLResourceStorageModePrivate}}, Float32, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, Nothing, Oceananigans.Fields.FieldBoundaryBuffers{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}}, Nothing, NamedTuple{(), Tuple{}}})
    @ Oceananigans.Models.HydrostaticFreeSurfaceModels ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/update_hydrostatic_free_surface_model_state.jl:26
 [38] HydrostaticFreeSurfaceModel(; grid::RectilinearGrid{Float32, Periodic, Periodic, Bounded, Float32, Float32, Float32, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, OffsetArrays.OffsetVector{Float32, Metal.MtlVector{Float32, Metal.MTL.MTLResourceStorageModePrivate}}, Oceananigans.Architectures.MetalBackend}, clock::Clock{Float32}, momentum_advection::Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}, tracer_advection::Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}, buoyancy::SeawaterBuoyancy{Float32, LinearEquationOfState{Float32}, Nothing, Nothing}, coriolis::Nothing, free_surface::ExplicitFreeSurface{Nothing, Float64}, forcing::NamedTuple{(), Tuple{}}, closure::Nothing, boundary_conditions::NamedTuple{(), Tuple{}}, tracers::Tuple{Symbol, Symbol}, particles::Nothing, biogeochemistry::Nothing, velocities::Nothing, pressure::Nothing, diffusivity_fields::Nothing, auxiliary_fields::NamedTuple{(), Tuple{}})
    @ Oceananigans.Models.HydrostaticFreeSurfaceModels ~/Documents/Projects/Oceananigans.jl/src/Models/HydrostaticFreeSurfaceModels/hydrostatic_free_surface_model.jl:196
 [39] top-level scope
    @ REPL[11]:1
 [40] top-level scope
    @ ~/.julia/packages/Metal/lnkVP/src/initialization.jl:57

I just realised that this isn't about integers but another float errro

[jagoosw]

Found where the promotion is occurring in the former which solves that error, and get us back to the FFT transform planning error that the nonhydostatic model gives

[jagoosw]

And if I modify the default free surface model to free_surface = ImplicitFreeSurface(gravitational_acceleration=eltype(grid)(g_Earth)) I also get the same error (the docstring is also wrong for what the default free surface model is).

[simone-silvestri]

for advection you have to specify the type if you do not pass the grid:

advection = Centered(Float32)
model = HydrostaticFreeSurfaceModel(; grid, tracer_advection = advection, momentum_advection = advection)

[jagoosw]

Ah yeah that also gives the fft error

[simone-silvestri]

you can also do Centered(grid) which will take care of it

[jagoosw]

Ooh so with:

using Oceananigans
using Oceananigans.BuoyancyModels: g_Earth

grid = RectilinearGrid(Oceananigans.Architectures.MetalBackend(), Float32; size = (3, 3, 50), extent=(10, 10, 500), halo = (3, 3, 3))

advection = Centered(Float32)

free_surface = ExplicitFreeSurface{Nothing, Float32}(nothing, Float32(g_Earth))

model = HydrostaticFreeSurfaceModel(; grid, tracer_advection = advection, momentum_advection = advection, free_surface)

time_step!(model, 1)

doesn't fail.

[simone-silvestri]

nice! Is it faster? BTW also ExplicitFreeSurface() should work because of the regularization we do inside the model construction

[jagoosw]

With a much bigger grid (512, 512, 512): Metal:

@btime time_step!(model, 1)
  629.676 ms (8811 allocations: 813.31 KiB)

(This is big enough to pin the Metal processor) CPU Float32 with 1 thread:

@btime time_step!(model, 1)
  4.373 s (1218 allocations: 339.55 KiB)

I ran it first with the grid above (10, 10, 50) and Metal was slower than CPU at 200ms vs micro seconds, but scaled much better.

nice! Is it faster? BTW also ExplicitFreeSurface() should work because of the regularization we do inside the model construction

Oh nice I didn't actually try that before.

[simone-silvestri]

Super nice, fast prototyping on laptops is incoming also for Mac users!

[jagoosw]

Ran the baroclinical adjustement example:

using Oceananigans
using Oceananigans.Units

Lx = 1000kilometers # east-west extent [m]
Ly = 1000kilometers # north-south extent [m]
Lz = 1kilometers    # depth [m]

grid = RectilinearGrid(Oceananigans.Architectures.MetalBackend(), Float32;
                       size = (48, 48, 8),
                       x = (0, Lx),
                       y = (-Ly/2, Ly/2),
                       z = (-Lz, 0),
                       topology = (Periodic, Bounded, Bounded))

advection = UpwindBiased(Float32)
free_surface = ExplicitFreeSurface()
coriolis = BetaPlane(Float32, latitude = -45)

model = HydrostaticFreeSurfaceModel(; grid,
                                      coriolis,
                                      buoyancy = BuoyancyTracer(),
                                      tracers = :b,
                                      momentum_advection = advection,
                                      tracer_advection = advection,
                                      free_surface)

This runs with none of the callbacks in about 4 seconds.

This output writer works (but TimeInterval won't work):

b = model.tracers.b
simulation.output_writers[:fields] = JLD2OutputWriter(model, (; b);
                                                      filename = filename * "_slice",
                                                      schedule = IterationInterval(10),
                                                      overwrite_existing = true)

And the timestep wizard also won't work because reductions cause a metal compiler error.

https://github.com/CliMA/Oceananigans.jl/assets/26657828/c75b33ba-5b06-454b-a883-93eb659d06d6

[glwagner]

If this will be important then we should streamline the user interface for changing to Float32. Ideally everything should take eltype from the grid I think without requiring tons of manual intervention.

[glwagner]

Where does the term "backend" come from in Oceananigans.Architectures.MetalBackend()? Shouldn't we call this Architecture? Or just MetalGPU perhaps.

[glwagner]

One idea to avoid a huge amount of refactoring could be to define something like

default_float_type = Ref(Float64)

Then to change to float32 we'd write

Oceananigans.default_float_type[] = Float32

Next, everywhere we've written FT=Float64 we change to FT=default_float_type[].

Basically that adds a global variable that users can modulate at the top of a script.

We'd also have to introduce type parameters here and there wherever Float64 was hardcoded (eg TimeInterval).

Note this is also a reason not to do clock updates with some kind of higher precision it would seem (which we have discussed a bit to solve some rounding errors)...

[jagoosw]

MetalBacked is what KA calls it but MetalGPU sounds better

[jagoosw]

I'm not sure why the TimeInterval thing is being passed into any kernels?

[glwagner]

MetalBacked is what KA calls it but MetalGPU sounds better

Well, we currently distinguish between KA backends and Oceananigans architectures. You could propose to eliminate that distinction; it's been debated before in the past. I think if we want to switch to using KA backend names, then we should also change "GPU" so we are consistent.

[glwagner]

Also I'm confused -- this is an issue? What code are we talking about? Is there a PR somewhere?

[glwagner]

Ooh so with:

using Oceananigans
using Oceananigans.BuoyancyModels: g_Earth

grid = RectilinearGrid(Oceananigans.Architectures.MetalBackend(), Float32; size = (3, 3, 50), extent=(10, 10, 500), halo = (3, 3, 3))

advection = Centered(Float32)

free_surface = ExplicitFreeSurface{Nothing, Float32}(nothing, Float32(g_Earth))

model = HydrostaticFreeSurfaceModel(; grid, tracer_advection = advection, momentum_advection = advection, free_surface)

time_step!(model, 1)

doesn't fail.

Where is this code? Can you link to the code rather than copy pasting. It's always better to link. Please try to do that.

[glwagner]

I'm not sure why the TimeInterval thing is being passed into any kernels?

I have no idea, but once there is a PR we can inspect the code. Without code its difficult to diagnose issues.

[jagoosw]

I have no idea, but once there is a PR we can inspect the code. Without code its difficult to diagnose issues.

I made a PR of my progress

[jagoosw]

Ooh so with:

using Oceananigans
using Oceananigans.BuoyancyModels: g_Earth

grid = RectilinearGrid(Oceananigans.Architectures.MetalBackend(), Float32; size = (3, 3, 50), extent=(10, 10, 500), halo = (3, 3, 3))

advection = Centered(Float32)

free_surface = ExplicitFreeSurface{Nothing, Float32}(nothing, Float32(g_Earth))

model = HydrostaticFreeSurfaceModel(; grid, tracer_advection = advection, momentum_advection = advection, free_surface)

time_step!(model, 1)

doesn't fail.

Where is this code? Can you link to the code rather than copy pasting. It's always better to link. Please try to do that.

This is just an MWE I was running

[glwagner]

Ooh so with:

using Oceananigans
using Oceananigans.BuoyancyModels: g_Earth

grid = RectilinearGrid(Oceananigans.Architectures.MetalBackend(), Float32; size = (3, 3, 50), extent=(10, 10, 500), halo = (3, 3, 3))

advection = Centered(Float32)

free_surface = ExplicitFreeSurface{Nothing, Float32}(nothing, Float32(g_Earth))

model = HydrostaticFreeSurfaceModel(; grid, tracer_advection = advection, momentum_advection = advection, free_surface)

time_step!(model, 1)

doesn't fail.

Where is this code? Can you link to the code rather than copy pasting. It's always better to link. Please try to do that.

This is just an MWE I was running

I think I was confused what MetalBackend meant or how / where this code ran. Now I see that it's defined in your PR, thank you for opening that.

[glwagner]

And if I modify the default free surface model to free_surface = ImplicitFreeSurface(gravitational_acceleration=eltype(grid)(g_Earth)) I also get the same error (the docstring is also wrong for what the default free surface model is).

Can you raise this in a separate issue?

[glwagner]

This output writer works (but TimeInterval won't work):

One possibility is that there is a type promotion occurring within align_time_step:

https://github.com/CliMA/Oceananigans.jl/blob/e16cdc6cfb67703df9e29368017868331f68b1c0/src/Simulations/run.jl#L24-L33

and

https://github.com/CliMA/Oceananigans.jl/blob/e16cdc6cfb67703df9e29368017868331f68b1c0/src/Simulations/run.jl#L133-L134

This would cause widespread issues in switching to single precision, so @simone-silvestri may want to take note.

The quick fix is to convert after calculating the aligned time-step:

Δt = aligned_time_step(sim, sim.Δt)
Δt = convert(eltype(sim.model), Δt)

I think we need a more defined interface for this in the long run though. We really need both eltype (the floating point type used by state variables, grid metrics, etc) and a timetype (the type of model.clock.time).

(Also I'm not sure eltype(model) is defined yet, but it should be...)