Model verification tests.

[ali-ramadhan]

This has been discussed quite a bit but beyond unit tests we also need model verification tests that ensure the model output is mathematically/scientifically correct by comparison with known analytic solutions or statistics.

The only one we've done so far is confirming the existence of steady-state convection rolls in Rayleigh–Bénard convection at Ra=5000 with an aspect ratio of 6. We can start with basic tests like isotropic diffusion and build up to more complicated stuff like slantwise convection.

@edoddridge has a lot of ideas regarding this, he might be a good person to talk to here.

[ali-ramadhan]

Some ideas by @christophernhill and @edoddridge

• Barotropic gyre (requires β effect).
• Frontal collapse.
• Chimney collapse.

[edoddridge]

To my mind there are three main ways to approach this:

1. design simulations such that the output can be directly compared with analytical solutions;
2. design simulations such that the statistics of the output can be compared with theory; or
3. design simulations to mimic published results

Option 1 restricts us to systems with tractable analytical solutions, but still contains a wealth of feasible problems, such as:

1. Munk gyre
2. Spin down of a flow-field under the influence of friction
3. Thermal wind balance: specify a density structure and compare model velocity fields with analytical solution
4. Rayleigh–Bénard convection (as mentioned previously)
5. Onset of baroclinic instability: compare growth rates with analytical predictions

Option 2 lets us explore dynamics in the presence of turbulence. Potential test cases here include:

1. 2D QG turbulence: explore energy and enstrophy cascades
2. lee wave generation and breaking (will require large-scale flow field and bathymetry)

Option 3 lets you do whatever you want - you just need to find a published result and try to reproduce it. It's unlikely that you'll get the exact same answer, so this option is more difficult to implement in a testing framework that doesn't require eyeballs to validate.

[glwagner]

What is the intent of these tests? Are they intended to be included in CI, or are they more in the style of “benchmarks” that are run relatively infrequently?

@edoddridge A twist on option 1 is to design a forcing that exactly cancels the terms associated with nonlinear and linear terms in a given equation for some simple initial condition consisting of sines and cosines. For example, pick an initial condition, calculate all terms in a given equation, and then write a forcing that exactly cancels those terms. Then check that the initial condition doesn’t change after a few time-steps. This method allows a test at low resolution with low computational burden and allows each nonlinear and linear term in each equation to be assessed separately.

It would also be good to run “benchmarks” that are designed to be run less frequently, which is a category I think some of the suggested tests fall into (?) Is the algorithm in Oceananigans.jl identical to some configuration of MITgcm? If so that opens the possibility to compare a solution grid-point for grid-point.

[edoddridge]

That's a good point @glwagner - I think it's a mix of CI tests and more involved experiments. Some of these can be easily integrated in to CI since they are very cheap to run. A one level wind-driven gyre should take only a few minutes of run time to equilibrate sufficiently.

Full simulations from option 3 are almost certainly too heavy for CI (unless there are plans afoot to use external resources for CI). These sort of simulations are more likely to be run occasionally and interrogated by real eyeballs. Having said that, you could setup CI to run a few time steps and compare the output with blessed output - this is what MITgcm does for its CI tests. This comes with a couple of advantages:

• the tests are useful setups for people to start using; and
• because they run regularly (for at least a few time steps) you know when the examples break

I like your idea of designing a forcing that exactly cancels the expected tendencies. It is a more rigorous test than "is the output the same as it was when I decided it was correct?".

Is the algorithm in Oceananigans.jl identical to some configuration of MITgcm? If so that opens the possibility to compare a solution grid-point for grid-point.

This might work, but you'll need to decide how closely it should match. You definitely won't get machine precision matches - we can't even do that with different MITgcm runs. The output from MITgcm depends on the machine, the compiler, and the optimisation level.

[glwagner]

Having said that, you could setup CI to run a few time steps and compare the output with blessed output - this is what MITgcm does for its CI tests.

Indeed --- the test of the Golden Master! That sounds like an excellent idea for Oceananigans.jl. No master is more golden than MITgcm.

This might work, but you'll need to decide how closely it should match. You definitely won't get machine precision matches - we can't even do that with different MITgcm runs. The output from MITgcm depends on the machine, the compiler, and the optimisation level.

Touche. I was naive.

[glwagner]

I'm going to start with some simple analytical solution tests (perhaps heat/salinity diffusion) until #73 is resolved. Then can begin on the 'designed forcing' CI tests.

[ali-ramadhan]

We now have some simple boundary condition physics test thanks to PR #118 and a couple of golden master tests in PR #140. #136 has some more specific physics tests that will be implemented.

We should either come up with some concrete goals for this issue to become resolvable, or we should close it and rely on it for inspiration in designing future tests.

[ali-ramadhan]

Closing as this issue is not resolvable but will serve as inspiration for future tests.

[edoddridge]

Are you planning to open targeted issues for (some of) the tests discussed here?

[ali-ramadhan]

Yes. We've implemented some simple tests recently (https://github.com/climate-machine/Oceananigans.jl/issues/126, https://github.com/climate-machine/Oceananigans.jl/pull/118, https://github.com/climate-machine/Oceananigans.jl/pull/140) and we're working on https://github.com/climate-machine/Oceananigans.jl/issues/136 but we should definitely do more.

Some tests will require a channel model which we're working towards (https://github.com/climate-machine/Oceananigans.jl/issues/100) but others that you've suggested should work in doubly periodic domains I think, e.g. spin down of a flow-field under the influence of friction, thermal wind balance, and Rayleigh–Bénard convection. Might be good to open issues for those.

[edoddridge]

Might be good to open issues for those.

Unless your brain is much better at remembering things than mine, I'd generally recommend doing that before closing the vague issue.

[ali-ramadhan]

Good point. I just have anxiety about the ever-increasing number of open issues :(

[glwagner]

My 2 cents: I think it’s fine to plan to refer back to this closed issue if, at some point in the future, we’d like to create many more verification tests.

Personally I think we should keep the number of verification tests from ballooning — they will become a burden to maintain as we evolve the API. This code is young! There’s a lot of progress to make.

[ali-ramadhan]

Woah, never thought I'd see the day that @glwagner advocates for fewer tests!

I think it's good to keep these things well-documented so I split these tests into three issues.

I agree we shouldn't implement all of these right now, so I'm going to create a far future milestone to categorize issues that we'd like to work on as the code matures.