Test the SSP timestepper

[valeriabarra]

The Climate Modeling Alliance

Software Design Issue 📜

Purpose

The purpose of this SDI is to test the new SSP 333 time-stepper in ClimaAtmos.

Cost/Benefits/Risks

Cost: @valeriabarra 's 60% time

Benefits: The time stepper, limiters, and vertical upwinding operators will be adjudged fit for purpose. Test coverage of the code will increase. Quality assurance will be performed and reported on.

Risks: Lack of simpler, incremental examples in the upstream repo. There is not currently a working spherical 3D simple example that uses the new SSP algorithm yet in ClimaTimeSteppers (only a 2D one). Documentation of the new interfaces is a bit lacking and scattered.

People and Personnel

• Lead: @valeriabarra
• Potential collaborators: @dennisYatunin , @szy21
• Reviewers: @dennisYatunin

Components

Part 1: Test SSP

In order for us to test the new SSP algorithm, we first need to

• [x] Upgrade ClimaTimeSteppers in ClimaAtmos from v0.6 to v0.7.1 (#1320)
• [x] Use the new ClimaODEFunction interface (no docstring, but see https://github.com/CliMA/ClimaTimeSteppers.jl/blob/main/src/functions.jl). (#1320)
• [x] Use the SSP333 algorithm. To do that, set the algorithm name (IMEXAlgorithm) and instead of algorithm_name as in the example, we will be using SSP333
• [x] Only horizontal advection and hyperdiffusion will be in the T_lim function (the order does not matter)
• [x] The other parts of the other tendencies (i.e., T_exp and T_imp will not change)

Part 2: Test different upwinding modes with SSP

• Assess the stability and results from moist integration tests. In order of complexity: Moist Baroclinic wave, Moist Held-Suarez, Aquaplanet with Radiation (use clearsky for radiation), first in the regular (coarse) pipeline config and then in the AMPI target resolution longrun one, each of these with the following configurations:
  • Config 1: Horizontal Limiters: OFF; Vertical 1st order upwinding: OFF
  • Config 2: Config 1 amended as follows: Horizontal limiters: ON
  • Config 3: Config 2 amended as follows: Vertical 1st order upwinding: ON for tracers
  • Config 4: Config 3 amended as follows: :zalesak vertical upwinding ON for tracers (instead of the 1st order one)
  • Config 5: Config 4 amended as follows: :zalesak vertical upwinding ON for tracers, energy and density.
  • Modified Config 5: Config 5 amended as follows: :zalesak vertical upwinding ON for tracers and energy. (added 04/07/23)

For Config 5, we need to modify the upwinding mode for density and add support for upwinding density (CLI option: --density_upwinding) with the different upwinding modes (similar to --energy_upwinding and --tracer_upwinding).

Part 3: Test different upwinding modes with SSP and topography

• Repeat each of the configurations above with the topography test cases

Part 2 consists of 3 different integration tests x 5 configurations each, without and with topography. That's a total of 30 experiments.

Inputs

Plan No. 102 from @cmbengue , plus additional inputs from @szy21 and others as needed.

Results and Deliverables

• The new SSP algorithm will be used in the GCM driver configuration and only after quality assurance and desired results are approved, it will be added to the CI pipeline.
• The dedicated jobs will be added to the CI pipeline(s) and their artifacts (including plots and quantitative results) will be used to assess quality and assurance that moisture is positive.
• Check that the simulation runs to completion and assess that moisture is positive for all time.
• Plots and information from the CI jobs will be produced to answer the questions posed.

Task Breakdown And Schedule

Part 1:

• [x] #1318
• [x] #1319
• [x] #1341

Part 2, no topography:

• [x] Config 1: #1440
• [x] Config 1: #1441
• [x] Config 2: Test SSP IMEX with limiter, with 1 Newton iteration and for different timestep
  • [x] Config 2': Test SSP IMEX with limiter, with 2 or 3 Newton iteration and for a certain timestep. See PR #1510
• [x] Config 3: #1503
• [x] #1513
• [x] #1519
• [x] Config 5: #1531
• [x] Modified Config 5: #1559
• [x] If instabilities/problems arise, open a new SDI to investigate the problem, identify problem, plan for a solution.

Part 3, with topography: (available when topography tests have been ironed out)

• [ ] Config 1: Test SSP IMEX without limiter, with the most stable configuration (2 or 3 Newton iterations), and for different timestep
• [ ] Config 2: Test SSP IMEX with limiter, with 1 Newton iteration and for different timestep
• [ ] Config 3: Test SSP IMEX with limiter, with the most stable configuration (2 or 3 Newton iterations), with vertical 1st order upwinding ON for tracers
• [ ] Config 4: Test SSP IMEX with limiter, with the most stable configuration (2 or 3 Newton iterations), using :zalesak vertical upwinding for tracers
• [ ] Config 5: Test SSP IMEX with limiter, with the most stable configuration (2 or 3 Newton iterations), using :zalesak vertical upwinding ON for tracers and energy.
• [ ] If instabilities/problems arise, open a new SDI to investigate the problem, identify problem, plan for a solution.

SDI Revision Log

@szy21

CC

@tapios @simonbyrne @cmbengue

Expected date of completion: March 13

[szy21]

Looks good, thanks! @tapios may have some comments on the flux correction. Tapio also mentions that we want to test it with topography, but that depends on when we have the topography runs in the ci. We should have them ready in 1-2 weeks, and can add those tests in this SDI or open a new one.

[tapios]

Yes, the 3rd-order upwinding should always include the flux correction. And we should include a topography test in all of this as soon as we have it.

[valeriabarra]

Yes, the 3rd-order upwinding should always include the flux correction.

Sorry @tapios , I need a clarification. We have different upwinding modes (each for upwinding energy and upwinding tracers). The flux-correction is one of the modes, but it is not included in the 3rd-order mode. Namely, we currently can specify --energy_upwinding and --tracer_upwinding and each of them can be either set to none (default, i.e. second-order), first_order , third_order, boris_book, or zalesak.

Do you want me to add another configuration to the configs above, basically: Config 5: Config 4 amended as follows: zalesak vertical upwinding ON (instead of the 3rd order one) ?

[tapios]

Config 4 should be 3rd-order upwinding with Zalesak flux correction for tracers (specific humidity).

It would be helpful to add a config 5 with 3rd-order upwinding (with flux correction) for density, energy, and tracers.

[szy21]

I forgot to add that Tapio mentioned that for simulations with source and sink of moisture (e.g. 0-moment microphysics), we need to have the relaxation timescale larger than timestep in order to ensure positivity.

[valeriabarra]

I forgot to add that Tapio mentioned that for simulations with source and sink of moisture (e.g. 0-moment microphysics), we need to have the relaxation timescale larger than timestep in order to ensure positivity.

Note to self: If needed, that would mean change the τ_precip in examples/hybrid/parameter_set.jl to use some τ_precip > dt.

[szy21]

Here is the list of figures we want (I will try to keep the list up-to-date):

For moist baroclinic waves (ref):

• lat-lon plots of 500 m vertical velocity, relative vorticity, temperature, and specific humidity at day 10
• lat-lon plots precipitation rates and veritical integrated water vapor, cloud water, and rain water at day 10

For moist Held-Suarez and aquaplanet simulations (ref for moist Held-Suarez)

• lat-pressure plots of zonal mean zonal wind, vertical velocity, temperature, specific humidity, relative humidty, and temperature variance
• plots of zonal mean precipitation, evaporation, sensible heat fluxes, and latent heat fluxes at the surface
• lat-lon snapshots of 850 hPa vertical velocity
• lat-lon snapshots of precipitation rates
• (I'm not sure what variables will show more clearly the difference between SSP and ARK, maybe @tapios has better suggesion?)
• For moist aquaplanet with radiation, let's just use clearsky to reduce the number of total configurations

[valeriabarra]

@tapios , for Config 2, we need a clarification. Do the horizontal limiters apply only to the tracers (ρq_tot and other tracers) or also to ρe_tot (which depends on q)?

[tapios]

Just tracers for now. (Later, we may want to test monotonicity, but not positivity, preserving limiters on energy too, but this doesn't need to be now.)

[valeriabarra]

Can anyone (@tapios and/or @cmbengue) formally approve this SDI for execution or request changes (if needed)? If anything different is expected, for instance (e.g. a separate script to produce results instead of the production of the results via our CI pipeline), please clarify. Thank you.

[tapios]

This looks good to me.

One further clarification: For 5), once we have upwinding for density and energy, we should also have horizontal limiters (monotonicity preserving) for them.

[valeriabarra]

Thanks @tapios . I need a further clarification.

In the hyperdiffusion tendency (please see these lines of code) we subtract the change in ρq_tot from both ρq_tot and ρ. Therefore, if we only limit ρq_tot it seems to me like we won't have the correct conservation of mass from ρ anymore. Is this understanding is correct? If so, we will have to discuss how to mitigate this. Thanks

[tapios]

Yes, this is to correct locally for mass tendencies owing to hyperdiffusion. (It would be better to call ᶜχρq_tot something like ᶜχq_tot, since it does not include the density factor.)

If you have the moisture tendency from the limiter, you can also add it in the continuity equation. If not, don't worry about it. It's just a local redistribution, which is tiny. It will not affect the integrated mass balance.

[valeriabarra]

As of today, 04/07, the tests without topography have been completed as requested.

Issues have been opened for the failing test cases:

• high-resolution, longruns Config 3: #1536
• Config 4: #1554
• Modified Config 5: #1562

[valeriabarra]

I was asked to compare the time-to-solution of SSP333 and ARS343 for the dry baroclinic wave test case, using the same resolution as in the Gardner paper (). I reported on the results in this comment in Issue #1573 . SSP333 is only 7% slower than ARS343 for the dry BW test case in consideration.

[charleskawczynski]

@valeriabarra, any updates on this issue?

[valeriabarra]

• Zalesak upwinding for MBW was added to the regular CI pipeline. I have reported results extensively in https://github.com/CliMA/ClimaAtmos.jl/issues/1562
• For the longruns pipeline, I have found configurations that work and make it long-time stable, but they have quite a small dt (they are also linked in https://github.com/CliMA/ClimaAtmos.jl/issues/1562) so for now we don't want to add it to the longrun pipeline.
• Daniel has opened a seprate issue with newly proposed strategies to try to reach stable results with higher dts (https://github.com/CliMA/ClimaAtmos.jl/issues/1679)
• Topology runs are not ready to be tested yet. I asked @akshaysridhar to keep me informed when they are. (I think the pending PR for that is https://github.com/CliMA/ClimaAtmos.jl/pull/1660)

[szy21]

Superseded by most recent issues on the timestepper.