search

COSIMA / access-om2

Deprecated ACCESS-OM2 global ocean - sea ice coupled model code and configurations.
22 stars 23 forks source link

Editing the tutorial on "Updating model bathymetry" #170

Open Acosta-Goncalves opened 5 years ago

Acosta-Goncalves commented 5 years ago

I'm following the "Updating model bathymetry" tutorial (https://github.com/COSIMA/access-om2/wiki/Tutorials#updating-restarts-for-new-bathymetry) and, as a newbie to this model, I'd like to contribute to the wiki by adding more detailed explanations so the tutorial is easier to follow for inexperienced people like me. Would that be ok?

aekiss commented 5 years ago

Sure, that would be great, thanks! Here's the existing text (in github markdown format). Make the changes you want (paste in here, enclosed between lines containing four backquotes) and I'll check them over and put them in.

## Updating restarts for new bathymetry 

If the MOM bathymetry file (topog.nc) needs to be changed on an existing run it's ocean restarts will need to be fixed up. Otherwise the restarts may not contain valid data at all points. This section describes a method for doing this. [This github issue comment](https://github.com/COSIMA/access-om2/issues/99#issuecomment-401532257) may also be helpful.

The approach taken is to create ocean restart files that match the new bathymetry (we call these the template restarts), then copy over all valid data from restarts for the existing run (call these the old restarts). The end result will be a restart that is the same as the existing run at all points which exist in both the old and the template (we call these the new restarts). Any new points that don't exist in the old restarts will contain whatever existed in the template restarts. This approach is very simple but does have a downside - if the bathymetry has changed a lot then there may be many points whose state is not consistent with the old restarts.

Step by step: 

1. Download topogtools. This contains a simple script that does the copying described above.

```{bash}
git clone https://github.com/COSIMA/topogtools.git

  1. You'll need template restarts from a run with the new bathymetry. If no previous run exists you'll need to create the restarts with a very short run from rest using the new topog.nc and matching CICE and MOM land masks kmt.nc and ocean_mask.nc as inputs (see step 7; create the land masks with topogtools/topog2mask.py topog.nc kmt.nc ocean_mask.nc).

  2. Collate the MOM restarts from both the template run with the new bathymetry and the run with the old bathymetry, e.g:

module use /g/data3/hh5/public/modules
module load conda/analysis3
payu collate -d template-run/archive/restart000/ocean
payu collate -d old-topo-run/archive/restart123/ocean
  1. Get an interactive PBS session with additional CPUs and memory, e.g.:
qsub -I -P e14 -v DISPLAY -q express -l ncpus=4,mem=128Gb,walltime=10:00:00
  1. Run the fix_restarts.py script to create new MOM restarts based on the old restarts but with the new bathymetry and any new ocean points filled in with the template, e.g.:
cd topogtools
fix_restarts.py --help
./fix_restarts.py template-run/archive/restart000/ocean old-topo-run/archive/restart123/ocean new-topo-run/archive/restart123/ocean

This can take quite a while. Note that fix_restarts.py requires Python 3 - you might need to do module use /g/data3/hh5/public/modules; module load conda/analysis3 first.

  1. Copy the restart files for the other model components, including the appropriate kmt.nc, e.g.:
cp old-topo-run/archive/restart123/accessom2_restart.nml new-topo-run/archive/restart123/
cp -r old-topo-run/archive/restart123/atmosphere new-topo-run/archive/restart123/
cp -r old-topo-run/archive/restart123/ice new-topo-run/archive/restart123/
cp template-run/archive/restart000/ice/kmt.nc new-topo-run/archive/restart123/ice
  1. Edit submodel input directories in config.yaml to ensure that MOM is using topog.nc and the matching land mask ocean_mask.nc, and that CICE is using the matching land mask kmt.nc. You might want to do payu setup to check that the resulting work directory links to the correct input files (then payu sweep to tidy up).

The configuration should now be ready to run.

aidanheerdegen commented 5 years ago

A wiki can be cloned, 

git clone git@github.com:COSIMA/access-om2.wiki.git

but there is no PR mechanism to submit changes back. It is possible to make the wiki it's own repository and use CI tools to update the wiki itself, but this is probably not a priority as there is not enough contributions to warrant it:

https://www.growingwiththeweb.com/2016/07/enabling-pull-requests-on-github-wikis.html

Acosta-Goncalves commented 5 years ago

Andrew, below the new version of the tutorial. (I might have further changes in the future)

## Updating restarts for new bathymetry 

To change the bathymetry on the ACCESS-OM2 model, you'll need to change the `topog.nc` input file, which contains the topography (or bathymetry in this case) of the model. The original `topog.nc` file _for the _1 degree model_ can be found at:
`/short/public/access-om2/input_rc/mom_1deg`
Similar folders exist for the 0.25 & 0.1 resolutions within the `input_rc` folder.

The `topog.nc` file does not contain any **lon-lat coordinates**, only depth data on a 360x300 grid. If lon-lat coordinates are needed to know where to modify the topography, they can be "imported" from any `ocean_grid.nc`output file. For example, it can be found at:
`/g/data/hh5/tmp/cosima/access-om2/1deg_jra55_ryf9091_kds50_july_bkd/output001/ocean`
The .nc file contains the `geolon_t` & `geolat_t` coordinates that can be copied into the `topog.nc` file for more precise topographic changes. On python for example, using xArrays, we could do something like:
```{bash}
ds_ocean_grid = xr.open_mfdataset('ocean_grid.nc')
ds_topog = xr.open_dataset('topog.nc')

ds_topog = ds_topog.assign_coords({'geolat_t':ds_ocean_grid.coords['geolat_t'].rename({'yt_ocean':'ny', 'xt_ocean':'nx'})})
ds_topog = ds_topog.assign_coords({'geolon_t':ds_ocean_grid.coords['geolon_t'].rename({'yt_ocean':'ny', 'xt_ocean':'nx'})})
ds_topog.coords['geolat_t'].values = ds_ocean_grid.coords['geolat_t'].values
ds_topog.coords['geolon_t'].values = ds_ocean_grid.coords['geolon_t'].values

It's worth looking at the OM2 model description paper by Kiss et al. 2019 for details on the grid (tripolar over 65N, increased resolution to 1/3 degree within 10 degrees of the Equator, and 1 degree Mercator elsewhere).

Since the input topog.nc is going to be changed, the CICE & MOM land masks will also need to be changed to match the new topography. The topogtools script (see below Step 1) will create the corresponding CICE mask file (kmt.nc) and MOM mask file (ocean_grid.nc).

If the MOM bathymetry file (topog.nc) needs to be changed on an existing run, its ocean restarts will need to be fixed up. Otherwise, the restarts may not contain valid data at all points. This section describes a method for doing this. This github issue comment may also be helpful.

The approach taken is to create ocean restart files that match the new bathymetry (we call these the template restarts), then copy over all valid data from restarts for the existing run (call these the old restarts). The end result will be a restart that is the same as the existing run at all points that exist in both, the old and the template (we call these the new restarts). Any new points that don't exist in the old restarts will contain whatever existed in the template restarts. This approach is very simple but does have a downside: if the bathymetry has changed a lot, then there may be many points whose state is not consistent with the old restarts.

Step by step:

  1. Download topogtools. This contains a simple script that does the copying described above. (See more info on topogtools here)
git clone https://github.com/COSIMA/topogtools.git

  1. You'll need template restarts from a run with the new bathymetry. If no previous run exists, you'll need to create the restarts with a very short run from rest. To achieve this, you'll need to create the aforementioned CICE & MOM land masks with topogtools by using the command: ./topog2mask.py topog.nc kmt.nc ocean_mask.nc Note that the above kmt.nc & ocean_mask.nc are the new mask files that will be replacing the older versions with the old bathymetry (i.e. the syntax of the command is: ./topog2mask.py input-topography-file output-cice-mask-file output-mom-mask-file) Then, make a copy in your working directory of the cice_1deg& mom_1deg folders located in /short/public/access-om2/input_236a3011/, and replace the 3 original .nc files with your new versions. Finally, specify in config.yaml the location of the new cice_1deg& mom_1deg folders, and the run from rest will be ready.

  2. Collate the MOM restarts from both the template run with the new bathymetry and the run with the old bathymetry, e.g:

module use /g/data3/hh5/public/modules
module load conda/analysis3
payu collate -d template-run/archive/restart000/ocean
payu collate -d old-topo-run/archive/restart123/ocean
  1. Get an interactive PBS session with additional CPUs and memory, e.g.:
qsub -I -P e14 -v DISPLAY -q express -l ncpus=4,mem=128Gb,walltime=10:00:00
  1. Run the fix_restarts.py script to create new MOM restarts based on the old restarts but with the new bathymetry and any new ocean points filled in with the template, e.g.:
cd topogtools
fix_restarts.py --help
./fix_restarts.py template-run/archive/restart000/ocean old-topo-run/archive/restart123/ocean new-topo-run/archive/restart123/ocean

This can take quite a while. Note that fix_restarts.py requires Python 3 - you might need to do module use /g/data3/hh5/public/modules; module load conda/analysis3 first.

  1. Copy the restart files for the other model components, including the appropriate kmt.nc, e.g.:
cp old-topo-run/archive/restart123/accessom2_restart.nml new-topo-run/archive/restart123/
cp -r old-topo-run/archive/restart123/atmosphere new-topo-run/archive/restart123/
cp -r old-topo-run/archive/restart123/ice new-topo-run/archive/restart123/
cp template-run/archive/restart000/ice/kmt.nc new-topo-run/archive/restart123/ice
  1. Edit submodel input directories in config.yaml to ensure that MOM is using topog.nc and the matching land mask ocean_mask.nc, and that CICE is using the matching land mask kmt.nc. You might want to do payu setup to check that the resulting work directory links to the correct input files (then payu sweep to tidy up).

The configuration should now be ready to run.

aekiss commented 5 years ago

Thanks @Acosta-Goncalves, I'll have a closer look later today.

In general a user should look at the ocean section config.yaml to see what input directory is being used for their run.

I noticed you're using /short/public/access-om2/input_rc/mom_1deg - bear in mind that _rc indicates this is a release candidate (i.e. a work in progress). It is currently unfinished and untested, and subject to change. That said, the topog.nc file won't be changed for this release.