How to run held-suarez model

[teopb]

Apologies for the basic questions. I am new to Clima but interested in running some simple held-suarez simulations to get started. I appreciate the example job scripts in the example folder, very useful! But I am not totally sure how to modify them properly to run Held-Suarez. I see some other examples for high top held-suarez.

Ideally I'd be trying to mimic the default CESM HS configuration as much as possible, which is T42, z30, mg17 resolution for the Eulerian spectral transform dynamical core. I'm not really sure how to set the dynamical core in Clima, is that by setting something in the driver or something in the "yml" file? I'm not totally bound to a specific dynamical core/solver so if its easier to make things run differently that's cool.

I see a sphere_held_suarez_rhotheta.yml, is that right? I don't see an associated toml file though. Is that because there are no additional parameters?

How can you pass in a specific set of IC's, I couldn't find much info on this?

I've read the readme on parameters, is it possible to alter existing parameters as part of the driver code by accessing the params of the integrator once it is set with the config?

Many, many thanks. Excited to try things out.

[szy21]

Thank you for your interest in ClimaAtmos. The model is still in a rapid development phase, and we should definitely improve our documentation. Here are some answers to your questions.

You would set most configurations in the yml file. The closest one might be longrun_hs_rhoe_dry_nz63_55km_rs35km.yml. For your specific configuration, I would suggest the following:

h_elem: 12
kappa_4: 3.0e16
z_elem: 30
z_max: 45000.0
dz_bottom: 200.0
dz_top: 3000.0
dt: "300secs"
forcing: "held_suarez"
t_end: [your simulation time]
output_dir: [your directory]

This should be similar to T42 z30. What does mg17 stand for? Note that I didn't try this myself. If there are numerical instabilities you can tune dt or kappa_4.

The initial condition is defined in this file. By default it uses DecayingProfile. If you have a simple function for the initial condition you can modify it there. The initial condition doesn't matter if you run a simulation to equilibrium and focus on the statistics.

It would be safer to change the parameters in the toml, following the documentation. That way it guarantees the same parameter is used throughout the code.

Hope this helps!

[charleskawczynski]

For a minimum working example, you could also check out our REPL scripts documentation. The closest job I can find to what you're looking for, there, is:

### Buildkite job `sphere_held_suarez_rhoe_hightop`

using Revise; import ClimaAtmos as CA;

config_dict = Dict();
config_dict["dz_bottom"] = 300.0;
config_dict["dt_save_to_disk"] = "4days";
config_dict["regression_test"] = true;
config_dict["t_end"] = "8days";
config_dict["forcing"] = "held_suarez";
config_dict["dt"] = "500secs";
config_dict["z_elem"] = 25;
config_dict["job_id"] = "sphere_held_suarez_rhoe_hightop";
config_dict["z_max"] = 45000.0;

config = CA.AtmosConfig(; config_dict);

include("examples/hybrid/driver.jl")

[teopb]

@szy21 and @charleskawczynski Thanks so much for the quick responses. Sorry for the delay on my end, crazy start to the semester.

@szy21 mg17 I guess defines the land/sea/ice mask. I guess that is probably not relevant here but is just a part of the resolution specification for CESM.

Thanks for pointing me to the initial conditions file. Is it possibly to start from a non-dynamic saved state? Like you would have to do if you wanted to restart the simulation from it's previous end point?

[szy21]

Thanks for pointing me to the initial conditions file. Is it possibly to start from a non-dynamic saved state? Like you would have to do if you wanted to restart the simulation from it's previous end point?

Unfortunately, currently there is no easy way to read the initial condition from a file. We can restart a simulation from the previous end point, but the file needs to be generated by a ClimaAtmos simulation.

[teopb]

Thanks again. Starting from previous ClimaAtmos simulation point is ok for my purposes. Based on another issue you made it seems like you can specify a frequency to save to jld2 files. Can you then restart from any saved point?

[szy21]

Yes, you can specify the frequency to save restart files (it's hdf5 now) with dt_save_restart. It should be able to restart from any saved point.

[teopb]

Thanks so much for your help. Is there any documentation or examples on the restart functionality you can point me to? I'd like to run a single long simulation (~500 days) for the generated initial conditions, then be able to start new simulations from the state at day 1, 2, 3, etc.

[szy21]

There is not much documentation right now, but you may find the configuration here helpful as an example. Specifically, the environment variable RESTART_FILE points to the restart file.

[teopb]

Thanks! I wasn't sure based on the link you sent how to specify what timeslice of the restart file you want to start from?

And to make the restart file you just specify something like dt_save_restart: "1days", wheres dt_save_to_disk: "1days" sets the netcdf save frequency?

[szy21]

Thanks! I wasn't sure based on the link you sent how to specify what timeslice of the restart file you want to start from?

And to make the restart file you just specify something like dt_save_restart: "1days", wheres dt_save_to_disk: "1days" sets the netcdf save frequency?

dt_save_restart sets the frequency of restart files, while dt_save_to_disk sets the frequency of diagnostic files (which includes more information than the restart files). They are both hdf5 files for now, and you should be able to start with either restart files or diagnostics files. To specify the restart file, you just give it the file name (with the full path) to the RESTART_FILE environment variable. e.g., in julia,

ENV["RESTART_FILE"] = ".../day1.0.hdf5"
ENV["RESTART_FILE"] = ".../day2.0.hdf5"

[teopb]

So it makes a different restart file for each save interval?

[szy21]

So it makes a different restart file for each save interval?

Yes.

[teopb]

Apologies for another question, but is there anyway to pass the restart file as an argument? As I said above, I am hoping to run many simulations from various restart points, so needing to update the environment variables every time is a bit of a hurdle.

[szy21]

Not for now unfortunately, but that's something I want to change at some point. Are you running simulations on a cluster? You may be able to update the environment variable automatically in a e.g. bash script.

[teopb]

Yes, running on a cluster. I will look into it more but I'm hoping to run multiple simulations in parallel. I may be able to launch a subshell for each one with the correct environment variable. Thanks again!

[teopb]

Having now successfully installed ClimaAtmos and having all tests passing I am trying to run a Held-Suarez configuration as you advised. Unfortunately I am erroring out. After warnings about the environment variables for hdf5 etc being deprecated the errors mostly seem to concern override_default_config(config_dict::. I have attached my error log. I apologize for all the questions but is hard to if I'm creating these files in the proper way based on current documentation.

Job Script

#!/bin/bash

#SBATCH --nodes=2
#SBATCH --ntasks=24
#SBATCH --job-name=clima_hs_base_run
#SBATCH --time=1:00:00
#SBATCH --output=clima_hs_base_simulation.log
#SBATCH --mail-user=teo.pricebroncucia@colorado.edu
#SBATCH --mail-type=ALL

module purge
module load gcc
module load openmpi
module load hdf5
module load julia/1.9.4

export JULIA_MPI_BINARY=system
export JULIA_NUM_THREADS=12
export CLIMACOMMS_CONTEXT="MPI"
export JULIA_HDF5_PATH=""

#export RESTART_FILE=$YOUR_HDF5_RESTART_FILE

CA_EXAMPLE='/projects/tepr6090/software/ClimaAtmos.jl/examples/'
DRIVER=$CA_EXAMPLE'hybrid/driver.jl'

julia --project=$CA_EXAMPLE -e 'using Pkg; Pkg.instantiate()'
julia --project=$CA_EXAMPLE -e 'using Pkg; Pkg.build("MPI")'
julia --project=$CA_EXAMPLE -e 'using Pkg; Pkg.build("HDF5")'
julia --project=$CA_EXAMPLE -e 'using Pkg; Pkg.API.precompile()'

mpiexec julia --project=$CA_EXAMPLE $DRIVER --config_file /projects/tepr6090/hs_uq/clima_hs/clima_hs_ground_truth.yml

YML file

h_elem: 12
kappa_4: 3.0e16
z_elem: 30
z_max: 45000.0
dz_bottom: 200.0
dz_top: 3000.0
dt: "300secs"
forcing: "held_suarez"
t_end: "1days"
output_dir: "/projects/tepr6090/clima_outputs"
dt_save_restart: "300secs"
toml: "/projects/tepr6090/hs_uq/clima_hs/parameters.toml"
diagnostics:
  - short_name: ta
    output_name: temp
    period: 300secs
    writer: nc
  - short_name: ua
    output_name: zonal_wind
    period: 300secs
    writer: nc
  - short_name: va
    output_name: meridional_wind
    period: 300secs
    writer: nc

toml

[potential_temp_vertical_gradient]
alias = "Δθ_z"
value = 10
type = "float"
description = "Potential temperature gradient with height. See Held and Suarez (1994) https://doi.org/10.1175/1520-0477(1994)075%3C1825:APFTIO%3E2.0.CO;2"

clima_hs_base_simulation.log

[szy21]

My guess is that the error is from how toml file is specified (see https://github.com/CliMA/ClimaAtmos.jl/issues/2167). Try to follow what we have here? @nefrathenrici would know more if this is the case.

[nefrathenrici]

Yes, the error is caused by an incorrect type in the configuration. Please put brackets around the toml file entry in the configuration file:

toml: ["/projects/tepr6090/hs_uq/clima_hs/parameters.toml"]

Please let me know if you run into other issues.

[szy21]

Closing this. Please feel free to reopen it if you have more questions.