search

CliMA / Oceananigans.jl

🌊 Julia software for fast, friendly, flexible, ocean-flavored fluid dynamics on CPUs and GPUs
https://clima.github.io/OceananigansDocumentation/stable
MIT License
937 stars 190 forks source link

contributing to Oceananigans #1153

Closed francispoulin closed 3 years ago

francispoulin commented 3 years ago

After discovering FourierFlows I guess it was only time before I came across Oceananigans. Great name by the way!

I would be interested in contributing to this and might even be able to get students to work on this as well. I am brand new to this repo and CliMA but am keen to learn more and work on something.

I have some ideas but also wonder what are the top 10 things that you want to have done now. If chatting would be easier, I'm happy to organize that sometime.

navidcy commented 3 years ago

After discovering FourierFlows I guess it was only time before I came across Oceananigans.

It's a slippery slope after FourierFlows.jl...

francispoulin commented 3 years ago

Thanks for the warning @navidcy . Both look great but I appreciate that they also have different goals.

Also, sorry for the very general question and maybe I should have been more specific.

I am very curious to learn about what is currently being worked on (and I can look at the issues for that I suppose) but one question that comes to mind is, can you solve the hydrostatic problem as well? I gather the incompressible sovler is for the more complicated non-hydrostatic case.

glwagner commented 3 years ago

@francispoulin ! No worries about the question. I like questions.

To answer your latest question --- our nonhydrostatic pressure solver is based on FFTs for regular grids and is therefore quite fast. In simple benchmarks long ago we found that other parts of the algorithm dominated the cost of a time-step. Because of that we aren't sure that having a hydrostatic-only solver would help. Of course, the story is different if we need to stretch the grid in horizontal directions.

That said, it would still be interesting to be able to solve hydrostatic-only problems. This would complicate the algorithm a bit because you have to distinguish the barotropic mode, and perform a 2D pressure solve. We already have a function to integrate the buoyancy field to obtain the hydrostatic pressure.

For "things I want worked on" my wish list falls into three categories: new physics features, new numerics / algorithms, and more boring software / UI work. Here's a couple...

  1. Finishing the vertically-stretched grid implementation (numerics). We started work on this but its incomplete. This is a tricky and arduous task but would be quite nice to have...

  2. Vertically-implicit time-stepping for diffusion terms (numerics). Ocean models typically use a time-stepping method that treats vertical diffusion terms implicitly. We haven't worried about this because we are focused on LES for the most part, or problems with very little diffusion. But ultimately we will need this, especially when we get around to implementing boundary layer closures. We have a tridiagonal solver that works on the GPU, so in some respects the hard work is already done for this problem.

  3. Closures for LES and ocean modeling (Dynamic Smagorinsky, Deardorff, k-epsilon, Gent-McWilliams, convective adjustment (?) etc --- physics). We have a need to implement new turbulence closures new and old alike. Gent-McWilliams is probably easy since we already have a Leith closure implemented which calculates the tensor needed to rotate the diffusivity into an isopycnal coordinate. The others are a bit more challenging.

  4. Extensive profiling and benchmarking on the GPU to identify bottlenecks / places for improvement in the algorithm (numerics). We might be able to make the code a lot faster (but we aren't sure).

  5. A solver for hydrostatic problems with a free surface that uses a split-explicit time-integration method. This is notoriously tricky, but we've made a lot of progress on this for the purposes of the ClimateMachine and it might be interesting to translate what we've learned into Oceananigans.

There's also some minor package stuff that I want to work on, such as refactoring the output writer (#963), improving logging / feedback while oceananigans builds a model or runs a simulation (#1013), getting arrays with named axes to work (#457), and figuring out what's going wrong with models that have Flat dimensions (#1024).

A more fun project is to come up with cool plotting recipes in 1D, 2D, and 3D (!) We'd like to show people how to use Makie (for example) to create really cool visualizations / animations of turbulence. The examples do some animation, but more complicated stuff (such as 3D contourf plots, or volume rendering) might require dedicated scripts.

@navidcy and I have also discussed projects to implement 1) a shallow water model using Oceananigans functionality and 2) a quasi-geostrophic model in Oceananigans (which would use the tridiagonal solver I mentioned earlier).

What else? The best project is probably one that aligns with you and your students' research... ? Definitely happy to chat further and also collaborate on something if you'd like.

ali-ramadhan commented 3 years ago

Definitely agree with everything @glwagner said!

I am very curious to learn about what is currently being worked on

I can share a bit about what I've been working on (and hope to work on).

Right now I'm almost done adding some support for Lagrangian particle tracking (PRs #1090 and #1091).

After that I'm hoping to upgrade the pressure solver to use 1D FFTs and DCTs for the CPU and GPU (#586, #1007). This would allow us to support all possible topologies on the GPU but might require some playing around with the CuFFT library (see https://github.com/JuliaGPU/CUDA.jl/issues/119). So if you know someone who's interested in FFTs and GPUs (or interested in learning about them!), this might be useful problem to work on! I think some other members of the Julia community would benefit from figuring this out as well.

Also hoping to revisit trying to support distributed parallelism with MPI (#590). So if there's someone interested in the nuts and bolts of distributed parallelism, might be great to work on this together!

We also have an early/experimental compressible version of Oceananigans.jl (see #1079 and https://github.com/ali-ramadhan/Atmosfoolery.jl) if you guys are interested in compressible fluid dynamics.

(and I can look at the issues for that I suppose)

We should definitely clean those up! We have a "good first issue" tag that should be useful in these situations but we should use the tag more often.

francispoulin commented 3 years ago

Thanks @glwagner and @ali-ramadhan for your suggestions. They all sound very interesting and gives me a lot to think about. Here are my initial reactions.

Given that I don't know the numerics of Oceananigans, yet, but know the physics, I can imagine myself contributing to the following pretty quickly:

  1. Shallow Water model (and maybe even a Green-Naghdi version) with several layers for fun.
  2. Layered and stratified Quasi-Geostrophic Model
  3. Hydrostatic Boussinesq Model (rigid lid first and then a free-surface)

Something that I can imagine giving to good undergraduate research students would be

  1. Verticall-Implicit time-stepping for diffusion.
  2. Profiling parallelism.

If I had keen student in computer science then maybe

  1. Distributed parallelism with MPI.

I have never used GPU's before but given the efficiency of GPU's vs CPU's that you have in the docs, I am definitely going to learn.

Initial thougths on 1 and 2: I know that FourierFlows has both QG and SW, which are spectrally based. I gather that Oceananigans is Finite Volume based. If I wanted to start with either of these would you suggest I start wtih FourierFlows or ignore that?

I kind of like the idea of having a QG code that can either be FV or Spectral, depending on the users whims, but that might be difficult, depending on how compatable these two libraries are.

I suppose I should start in reading the code in detail and planning out what I think needs to be done. Any suggestions are welcome!

glwagner commented 3 years ago

I think any of those projects would be super welcome. A 1-layer shallow water model would be nice and simple, and @ali-ramadhan could use it for his MPI parallelization work since it has no pressure solver.

A hydrostatic Boussinesq model would be super useful too of course. I think we should talk. We are working on a similar model using discontinuous Galerkin numerics over at ClimateMachine.jl and these projects would have overlapping goals. Nevertheless I would personally find it extremely useful to be able to run Oceananigans.IncompressibleModels with a HydrostaticPressureSolver and ExplicitFreeSurface since I could use it to benchmark / compare with the DG functionality that I'm working on for ClimateMachine, and the relative advantages of high-order finite volume methods compared with DG may still be in question, at least for structured grids (meaning that an Oceananigans development effort is not a waste of time). Perhaps a zoom conversation on that topic is in order.

I agree with the framework / level-of-difficulty that you've laid out for student projects!

As for deciding FourierFlows vs Oceananigans, that's up to you for sure and depends basically on whether you are okay staying with doubly-periodic systems (what FourierFlows currently supports) or whether you want to do problems with boundaries (what Oceananigans promises). FourierFlows is a nice tool for idealized problems and theoretical work (and you can run pretty large problems these days since it has GPU support); Oceananigans is evolving more towards a tool for ocean modeling that is capable of doing idealized problems, but flexible enough to use for large eddy simulation and realistic non-hydrostatic modeling, in bounded domains and hopefully irregular domains as well, eventually, using immersed boundaries.

Switching between the two would be a dream... ! I'm afraid this is too futuristic for Oceananigans. It may be possible to build FV functionality into FourierFlows (which is generally more flexible than Oceananigans, but has fewer features for the specific problem of the nonhydrostatic Boussinesq equations), but that's not a small piece of work.

francispoulin commented 3 years ago

Okay, then let's start with a one-layer, reduced-gravity SW model with a free-surface. Afterwards, we can get fancy and do a rigid lid version if we like and then talk about other interesting problems. I would be happy to do it in both but let's focus on Oceananigans as that allows a greater versatility in terms of the boundary conditions.

I am also interested in solving eigenvalue problems to compute stability characteristics in Julia and have done a bit of this already. 1D problems are of course easy to solve but 2D ones are much trickier. It would be great to talk to a Julia expert about what best practices they might suggest.

I am happy to setup a zoom conversation sometime. I don't know what time-zone everyone is in but if there is a typical time that might meet please suggest it and I will see if I can make that time.

I'm also happy to close this issue whenever you like as an idea as come up.

navidcy commented 3 years ago

ShallowWaterModels were kickstarted. I'll close this for now and anybody should feel free to open to continue chatting :)