Error: InvalidIRError: compiling MethodInstance for OceanBioME.Light.gpu_update_TwoBandPhotosyntheticallyActiveRadiation!

[vtamsitt]

Hi!

I'm attempting to test GlobalOceanBioME on a gpu, and am interested in doing some test runs and validation of the global model using LOBSTER with carbonate chemistry, eventually at higher resolutions (and contributing to development/improvements to this, happy to discuss what would be useful to contribute).

I am getting the error uploaded in this file error_27022024.log building the simulation when calling OneDegreeSurfacePAR(). Possibly this could be a compatibility issue running with newer versions of OceanBioME and CUDA? Note that if I remove the surface_photosynthetically_active_radiation kwarg and just run with the constant default_surface_PAR instead, the model will run (although I get NaNs in the nitrate field after ~8 days...).

My script is here

Environment:

• Julia: v1.10.0
• OceanBioME v0.10.0
• Oceananigans v0.90.8
• CUDA v5.1.2

Thanks for your help! Veronica

[navidcy]

Hi @vtamsitt!

is there a reason you don’t want to use the version of the packages in the toml files in the repo? Your problem indeed probably stem from breaking changes that were introduced by minor releases on OceanBioME/Oceananigans that came after this repo was created.

[vtamsitt]

Hi @navidcy!

Yeah good question, I did try it first with the versions in the toml file and also had issues, but that could have been user error, I'm learning Julia as I go here. I'll go back and check that again and see if it resolves this issue.

[navidcy]

We can zoom and I can help you with Julia to try to reproduce the installation that the repo suggests. Email me and we can arrange.

[navidcy]

If then we need to update the code to move to newer versions of the packages we can also do that later.

[vtamsitt]

Thanks Navid! I'll shoot you an email.

[navidcy]

@jagoosw, I see ClimaOcean in the Manifest.toml but not in the Project.toml... strange... how come?

[navidcy]

I'm reopening this to wait to hear from @vtamsitt.

@vtamsitt, for me, the script runs just fine on my laptop when I use architecture = CPU(). I do get some arch_array-related warnings but that's fine. The next release of OceanBioME will deal with those warnings; https://github.com/OceanBioME/OceanBioME.jl/pull/171

julia> include("scripts/one_degree_proof_of_concept.jl")
[ Info: Oceananigans will use 8 threads
boundary_layer_turbulence_closure = ConvectiveAdjustmentVerticalDiffusivity{VerticallyImplicitTimeDiscretization}(background_κz=0.0 convective_κz=0.1 background_νz=0.01 convective_νz=0.0)
[ Info: Reading initial conditions...
[ Info: ... read initial conditions (22.760 ms)
[ Info: Reading boundary conditions...
[ Info: ... read boundary conditions (252.499 ms)
┌ Warning: `arch_array` is deprecated. Use `on_architecture` instead.
└ @ Oceananigans.Architectures ~/.julia/packages/Oceananigans/ctYdG/src/Architectures.jl:131
┌ Warning: `arch_array` is deprecated. Use `on_architecture` instead.
└ @ Oceananigans.Architectures ~/.julia/packages/Oceananigans/ctYdG/src/Architectures.jl:131
┌ Warning: `arch_array` is deprecated. Use `on_architecture` instead.
└ @ Oceananigans.Architectures ~/.julia/packages/Oceananigans/ctYdG/src/Architectures.jl:131
┌ Warning: `arch_array` is deprecated. Use `on_architecture` instead.
└ @ Oceananigans.Architectures ~/.julia/packages/Oceananigans/ctYdG/src/Architectures.jl:131
┌ Warning: `arch_array` is deprecated. Use `on_architecture` instead.
└ @ Oceananigans.Architectures ~/.julia/packages/Oceananigans/ctYdG/src/Architectures.jl:131
┌ Warning: `arch_array` is deprecated. Use `on_architecture` instead.
└ @ Oceananigans.Architectures ~/.julia/packages/Oceananigans/ctYdG/src/Architectures.jl:131
┌ Info: Created 360×150×48 ImmersedBoundaryGrid{Float64, Periodic, Bounded, Bounded} on CPU with 5×5×5 halo:
│ ├── immersed_boundary: GridFittedBottom(mean(z)=-2482.07, min(z)=-9021.22, max(z)=100.0)
│ ├── underlying_grid: 360×150×48 LatitudeLongitudeGrid{Float64, Periodic, Bounded, Bounded} on CPU with 5×5×5 halo and with precomputed metrics
│ ├── longitude: Periodic λ ∈ [-180.0, 180.0) regularly spaced with Δλ=1.0
│ ├── latitude:  Bounded  φ ∈ [-75.0, 75.0]   regularly spaced with Δφ=1.0
└ └── z:         Bounded  z ∈ [-5244.5, 0.0]  variably spaced with min(Δz)=10.0, max(Δz)=410.5
┌ Warning: `ImmersedBoundaryCondition` is experimental.
└ @ Oceananigans.ImmersedBoundaries ~/.julia/packages/Oceananigans/ctYdG/src/ImmersedBoundaries/immersed_boundary_condition.jl:59
┌ Warning: `ImmersedBoundaryCondition` is experimental.
└ @ Oceananigans.ImmersedBoundaries ~/.julia/packages/Oceananigans/ctYdG/src/ImmersedBoundaries/immersed_boundary_condition.jl:59
[ Info: Building a model...
┌ Warning: `arch_array` is deprecated. Use `on_architecture` instead.
└ @ Oceananigans.Architectures ~/.julia/packages/Oceananigans/ctYdG/src/Architectures.jl:131
┌ Warning: `ImmersedBoundaryCondition` is experimental.
└ @ Oceananigans.ImmersedBoundaries ~/.julia/packages/Oceananigans/ctYdG/src/ImmersedBoundaries/immersed_boundary_condition.jl:59
┌ Warning: `ImmersedBoundaryCondition` is experimental.
└ @ Oceananigans.ImmersedBoundaries ~/.julia/packages/Oceananigans/ctYdG/src/ImmersedBoundaries/immersed_boundary_condition.jl:59
warning: /Users/navid/.julia/packages/KernelAbstractions/zPAn3/src/extras/loopinfo.jl:28:0: loop not unrolled: the optimizer was unable to perform the requested transformation; the transformation might be disabled or specified as part of an unsupported transformation ordering
┌ Info: ... built HydrostaticFreeSurfaceModel{CPU, ImmersedBoundaryGrid}(time = 0 seconds, iteration = 0)
│ ├── grid: 360×150×48 ImmersedBoundaryGrid{Float64, Periodic, Bounded, Bounded} on CPU with 5×5×5 halo
│ ├── timestepper: QuasiAdamsBashforth2TimeStepper
│ ├── tracers: (N, P, Z, D, T, S)
│ ├── closure: Tuple with 4 closures:
│ │   ├── HorizontalScalarDiffusivity{ExplicitTimeDiscretization}(ν=50000.0, κ=(N=0.0, P=0.0, Z=0.0, D=0.0, T=0.0, S=0.0, e=50000.0))
│ │   ├── ConvectiveAdjustmentVerticalDiffusivity{VerticallyImplicitTimeDiscretization}(background_κz=0.0 convective_κz=0.1 background_νz=0.01 convective_νz=0.0)
│ │   ├── VerticalScalarDiffusivity{VerticallyImplicitTimeDiscretization}(ν=PiecewiseConstantVerticalDiffusivity(-49.0, 0.0001, 0.0001), κ=(N=1.0e-5, P=1.0e-5, Z=1.0e-5, D=1.0e-5, T=1.0e-5, S=1.0e-5))
│ │   └── IsopycnalSkewSymmetricDiffusivity(κ_skew=900.0, κ_symmetric=900.0)
│ ├── buoyancy: SeawaterBuoyancy with g=9.80665 and BoussinesqEquationOfState{Float64} with ĝ = NegativeZDirection()
│ ├── free surface: ImplicitFreeSurface with gravitational acceleration 9.80665 m s⁻²
│ │   └── solver: MatrixImplicitFreeSurfaceSolver
│ ├── advection scheme:
│ │   ├── momentum: Vector Invariant, Dimension-by-dimension reconstruction
│ │   ├── N: WENO reconstruction order 5
│ │   ├── P: WENO reconstruction order 5
│ │   ├── Z: WENO reconstruction order 5
│ │   ├── D: WENO reconstruction order 5
│ │   ├── T: WENO reconstruction order 5
│ │   └── S: WENO reconstruction order 5
└ └── coriolis: HydrostaticSphericalCoriolis{Oceananigans.Coriolis.ActiveCellEnstrophyConserving, Float64}.
[ Info: Model building time: 4.390 minutes
┌ Warning: model.closure contains a function somewhere in its hierarchy and will not be checkpointed.
└ @ Oceananigans.OutputWriters ~/.julia/packages/Oceananigans/ctYdG/src/OutputWriters/checkpointer.jl:88
[ Info: Running a simulation with Δt = 20 minutes from 345 days until 346 days
[ Info: Initializing simulation...
[ Info: 345 days in 0 seconds with Δt = 20 minutes
[ Info:     ... simulation initialization complete (16.543 seconds)
[ Info: Executing initial time step...
[ Info:     ... initial time step complete (39.227 seconds).
[ Info: 345.139 days in 1.355 minutes with Δt = 20 minutes
[ Info: 345.278 days in 1.772 minutes with Δt = 20 minutes
[ Info: 345.417 days in 2.184 minutes with Δt = 20 minutes
[ Info: 345.556 days in 2.607 minutes with Δt = 20 minutes
...

[jagoosw]

Just to add, I've been trying to run the script with the NPZD model and keep getting something going wrong at around day 520. I think it's a combination of how I'm initialising causing phytoplankton to rapidly die off in some areas, and how its dealing with negatives but I'll get back to you when I've worked it out.

[jagoosw]

Also, when I ran the proof of concept one for the paper I took more of a brute force approach of resetting tracers and tendencies which were NaN to zero.

Another separate thought I had was that it might be worthwhile asking someone like Simone for the physics fields from a run he's done so it will have warmed up and have had more attention paid to the setup. And then you could use them to prescribe the velocity and temperature fields.

[vtamsitt]

@navidcy thanks! it runs fine on my laptop.

@jagoosw ok that would be great to know when you've figured out the NPZD issues. I agree that it would be a good idea to test the BGC with some physics fields that are more spun up and better setup so it's easier to test/evaluate the BGC model(s).

[navidcy]

nice to hear! I'm closing the issue then

[jagoosw]

I've checked and it will run now. These are some videos of it running for ~2 years:

Phytoplankton:

https://github.com/OceanBioME/GlobalOceanBioME/assets/26657828/843c1575-0b20-4f89-89f7-5db3fcff1741

Temperature:

https://github.com/OceanBioME/GlobalOceanBioME/assets/26657828/f9afdfa3-b9a7-47e9-b450-b66b4d3db4e4

PAR:

https://github.com/OceanBioME/GlobalOceanBioME/assets/26657828/f5708beb-2709-4034-a779-a1204ac07527

I'm not sure but I think the phytoplankton looks a bit weird because even the lower PAR levels are enough for it to grow so it's more following the temperature.

I will try and get hold of some spun up physics.

[jagoosw]

Btw what got it to work was this PR: https://github.com/OceanBioME/OceanBioME.jl/pull/170 so that does need to be merged before it might work here

[navidcy]

PAR looks like it's growing on land?

We should include one/more of these videos in the repo's README!

[jagoosw]

I think the PAR goes on the land because the satellites measure it from lakes etc. too and at a higher resolution, and because of how I interpolated it the box is just filled the whole cell if there are any measurements available. We perhaps should modify https://github.com/OceanBioME/OceanBioME.jl/blob/cec152b82d6bf89f6061e786e1fabb2cb32165f6/src/Light/2band.jl#L1-L37 to not calculate for land cells at some point.

[jagoosw]

I'll make a PR to add the video

[vtamsitt]

Nice videos, yeah it would be good to add to the README. I can take a look at modifying the PAR calculation to not calculate land cells when I have time.