GPU/IBM error with Centered Derivatives

[whitleyv]

I want to add ϵ = Field(IsotropicPseudoViscousDissipationRate(model)) to the outputs of my simulation, but I'm getting unsupported dynamic function invocation errors related to the centered derivatives used with an IBM grid on a GPU when I run it (full output here: MEWout.txt).

This is using the main Oceanostics branch with Oceananigans v0.77.5 and CUDA v3.9.1. It works without the IBM and on CPU.

I've tried being a little more explicit in which modules/ functions I needed, but no luck yet.

using Oceanostics: IsotropicPseudoViscousDissipationRate
using Oceananigans.Operators
using Oceananigans.ImmersedBoundaries

[tomchor]

Sorry I took a while to get to this. Busy day!

I can't see anything here that jumps to the eye. The error is a bit obscure to me. This function is working for me on GPU using CUDA 3.9. I'm assuming you're running on Julia 1.6, right?

Can you write a minimum working example so that I can reproduce the error?

Something you can do in the mean time is to try a different CUDA version, although I don't know if that'll solve it.

[glwagner]

The line is here:

https://github.com/CliMA/Oceananigans.jl/blob/31ffa28e11c9a4f6b1e44296f742edd54067582b/src/ImmersedBoundaries/conditional_derivatives.jl#L37

@whitleyv maybe this will help: https://github.com/CliMA/Oceananigans.jl/pull/2818

[whitleyv]

I'm using Julia 1.7.1, and I tried an older version of CUDA with no luck. MWE below:

using CUDA: has_cuda_gpu
using Oceananigans
using Oceananigans.Operators
using Oceananigans.ImmersedBoundaries
using Oceanostics: IsotropicPseudoViscousDissipationRate

underlying_grid = RectilinearGrid(GPU(); size = (5, 5, 5), extent=(1,1,1), halo = (4, 4, 4))

@inline is_immersed(x, y, z) = z < -.2
immersed_grid = ImmersedBoundaryGrid(underlying_grid, GridFittedBoundary(is_immersed))

kwargs_model = (closure = SmagorinskyLilly(),
              advection = WENO(),
                coriolis = FPlane(f=1.0),
            timestepper = :RungeKutta3)

model = NonhydrostaticModel(grid = immersed_grid,; kwargs_model...)

simulation = Simulation(model, Δt = 0.2, stop_time = 1.0)

ϵ  = Field(IsotropicPseudoViscousDissipationRate(model))
outputs = merge(model.velocities, (; ϵ=ϵ,))

simulation.output_writers[:fields] = JLD2OutputWriter(model, outputs;
           schedule = TimeInterval(0.5),
           filename = "MEWccc",
           overwrite_existing = true)

run!(simulation)

@whitleyv maybe this will help: CliMA/Oceananigans.jl#2818

I'll try this, though I'm only using Oceananigans v0.77.5 right now, since I can't update to Julia 1.8, yet.

[tomchor]

@whitleyv maybe this will help: CliMA/Oceananigans.jl#2818

I'll try this, though I'm only using Oceananigans v0.77.5 right now, since I can't update to Julia 1.8, yet.

You can try Julia 1.8 from the precompiled binaries. It's the one I've been using on Casper to test things out when I need to test newer branches. It's not optimized for the server, but so far it's worked for everything I needed and with minimal downside on performance.

[tomchor]

@whitleyv is this still an issue or should we close it? It sounds like updating Oceananigans solved it.

[whitleyv]

Yes, updating Oceananigans fixed the issue!