CliMA / Oceananigans.jl

🌊 Julia software for fast, friendly, flexible, ocean-flavored fluid dynamics on CPUs and GPUs
https://clima.github.io/OceananigansDocumentation/stable
MIT License
987 stars 193 forks source link

Interface for allocating the prognostic fields and diagnostic fields at the same time #2509

Open glwagner opened 2 years ago

glwagner commented 2 years ago

Efficiently distributing a model (especially across GPUs) requires minimizing the number of messages. update_state!(model) looks like:

  1. Fill halos for the prognostic variables (requires message passing)
  2. Computation of diagnostic variables
  3. Fill halos for diagnostic variables (requires message passing)

We can thus minimize message passing by allocating two buffers: one for the prognostic fields, and one for the diagnostic fields.

@simone-silvestri and I propose a new utility field_tuple for building tuples of fields to solve this.

francispoulin commented 2 years ago

This sounds interesting but I'm not sure if I complete understand.

Can we think of an example of a prognistic field as velocity: 𝐯, and a diagnostic field as vorticity: ζ?

If yes, I see that you need to fill halos to evolve the velocity and then you need to fill halos when you compute the vorticity. Are you simply saying we allocate the space for the two sets of halos at the same time?

glwagner commented 2 years ago

Yes, except that we don't allocate memory for vorticity at all; we compute it on the fly.

The prognostic fields are velocity components and tracers. For hydrostatic, w is diagnostic and free surface is prognostic. For nonhydrostatic, w is prognostic.

The diagnostic fields include pressure, w if hydrostatic, and any diagnostics fields associated with the closure.

We will also have a user interface for closure prognoetic fields at some point too.