f.e23.FWmaHIST.ne0np4.tromso01.ne30x8.test001

[marcin-kupilas]

Testing tromso grid

Grid already created, including grid files (SCRIP, LATLON, MESH, EXODUS)

Running from 1980-01-01

Storage

• Instructions: https://wiki.ucar.edu/display/MUSICA/Custom+Grid+in+CESM3#CustomGridinCESM3-InputfilesGenerateCESMinputfilesonnewgrid

export VRM_CESM_TAG=/glade/derecho/scratch/mmkupilas/tag_cam6_4_028/
export VRM_REPO_PATH=/glade/derecho/scratch/mmkupilas/RR/Grids/
export VRM_GRID_TAG=tromso01
export VRM_GRID_NAME=ne0np4.tromso01.ne30x8
export VRM_GRID_NCOL=79274
export VRM_DATE=240820 # Format YYMMDD 20th August 2024 

---

GRID PATH /glade/derecho/scratch/mmkupilas/RR/Grids/ne0np4.tromso01.ne30x8

Create input data

---

Regrid CAM IC file on casper

Scripts to from ne30 to refined grid are in https://github.com/NCAR/IPT/tree/master/Initial_conditions

I downloaded them to /glade/derecho/scratch/mmkupilas/RR/IPT/Initial_conditions

Using Peter's IC /glade/campaign/cgd/amp/pel/inic/L135_ne30pg3_ne30pg3_mg17.cam.i.1979-01-11-00000.nc

--------------------------- Second attempt

Outcome: I have re-gridded Peter's IC file onto the tromso01 grid. Not all variables look ok. Some variables have discontinuities or strange nan-numbers, e.g., O, CO, CO2, H. Values of 1e305. Is this a problem? Or is this just an artefact of re-gridding and things should smooth out as you run?

Try to run it for a few days and see if that happens. It's what happened in MG.

See below for paper trail:

scripts:

• gen_CAMncdata (prefers FV -> SE)

• IPT/ic

• [x] Generate 135 level grid from a 110 level FV input file

  • For details see the following file (user changes). If want to see changes, diff with original. /glade/derecho/scratch/mmkupilas/RR/Community_Mesh_Generation_Toolkit/VRM_tools/gen_CAMncdata/ne0np4.tromso01.ne30x8/interpic_script_tromso01-2024-09-30.sh

• [x] Use grid from 1 as target grid to re-grid Peter's IC file

  • Existing file with new grid: /glade/derecho/scratch/mmkupilas/RR/Grids/ne0np4.tromso01.ne30x8/inic/ne0np4.tromso01.ne30x8_L135_1979-01-11_c240820.nc
  • Failed : There is no lat/lon data in Peter's IC file, but there is lat_d/lon_d. Note - dimension ncol is called ncol_d...
  • Options:
    • 1. Rename dimensions and variables.
         • Worked (for now)
    • 2. Find lat lon data from another ne30 grid and put into IC file?
    • 3. Run my ne30 model for a timestep and output an IC file
    • 4. Use an h0 file from my ne30 run?
         • Failed - my data is interpolated prior to writing out, so won't work as an IC file. My h0 files only contain 29 variables also.

---------------------------

--------------------------- First failed attempt

• [ ] Compile interpic_new in /glade/derecho/scratch/mmkupilas/tag_cam6_4_028/tools/interpic_new
• [ ] Interpolate Peter's IC file
  • I used gen_CAMncdata to regrid Peter's IC file. I got an error in the log file saying ERROR required variable US, but the new IC file has been created. It contains 120 variables, same as Peter's IC file - so maybe we're ok.
  • No we are not ok - it produced an IC file where some variables have new ncol and others have old ncol. Go again.

----------------------------

END regrid CAM IC file

---

Regrid 'atmsrf' file The resulting file is assigned to drydep_srf_file in user_nl_cam. on casper

---

Create Topography file on casper

Re-gridded - /glade/campaign/cgd/amp/pel/topo/cubedata/gmted2010_modis_bedmachine-ncube3000-220518.nc

• cd /glade/derecho/scratch/mmkupilas/RR/Grids/ne0np4.tromso01.ne30x8
• cd topo
• git clone https://github.com/NCAR/Topo.git Topo
• cd Topo
• cd cube_to_target
• module load gcc
• module list
• gmake -f Makefile clean
• gmake -f Makefile

Run: qcmd -l walltime=12:00:00 -l select=1:ncpus=1 -A P93300043 -- ./cube_to_target --rrfac_manipulation --grid_descriptor_file='/glade/derecho/scratch/mmkupilas/RR/Grids/ne0np4.tromso01.ne30x8/grids/tromso01_np4_SCRIP.nc' --intermediate_cs_name='/glade/campaign/cgd/amp/pel/topo/cubedata/gmted2010_modis_bedmachine-ncube3000-220518.nc' --output_grid='tromso01' --rrfac_max=8 --smoothing_scale=100.0 -u 'Marcin Kupilas, m.m.kupilas@leeds.ac.uk' > out_mmk

topo file tromso01_gmted2010_modis_bedmachine_nc3000_Laplace0100_noleak_20241001.nc created in /glade/derecho/scratch/mmkupilas/RR/Grids/ne0np4.tromso01.ne30x8/topo/Topo/cube_to_target/output

moved to /glade/derecho/scratch/mmkupilas/RR/Grids/ne0np4.tromso01.ne30x8/topo/tromso01_gmted2010_modis_bedmachine_nc3000_Laplace0100_noleak_20241001.nc

---

Generate CTSM (CLM) surface datasets on derecho

• cd /glade/derecho/scratch/mmkupilas/RR/Grids/ne0np4.tromso01.ne30x8

• git clone https://github.com/ESCOMP/CTSM ctsm5.2.007

• cd ctsm5.2.007

• git checkout ctsm5.2.007

• bin/git-fleximod update Build executable for mksurfdata

• cd ./tools/mksurfdata_esmf

• ./gen_mksurfdata_build Generate namelist files containing specifications for creating the surface datasets.

First load the conda environment:

• module load conda/latest
• conda activate npl Run the make namelist script

(npl) - /glade/derecho/scratch/mmkupilas/RR/Grids/ne0np4.tromso01.ne30x8/ctsm5.2.007/tools/mksurfdata_esmf/gen_mksurfdata_namelist --res tromso.ne30x8 --start-year 1979 --end-year 2026 --ssp-rcp SSP3-7.0 --model-mesh /glade/derecho/scratch/mmkupilas/RR/Grids/ne0np4.tromso01.ne30x8/grids/tromso01_np4_MESH.nc --model-mesh-nx 79274 --model-mesh-ny 1

Now create a job script (use your namelist filename):

• /glade/derecho/scratch/mmkupilas/RR/Grids/ne0np4.tromso01.ne30x8/ctsm5.2.007/tools/mksurfdata_esmf/gen_mksurfdata_jobscript_single --number-of-nodes 4 --tasks-per-node 128 --namelist-file surfdata_tromso01.ne30x8_SSP3-7.0_1979_78pfts_c241001.namelist

This creates mksurfdata_jobscript_single.sh. Edit this file to use your project # in '#PBS -A P...'.

• qsub mksurfdata_jobscript_single.sh This creates the netcdf files that will be used in your simulation, specified in user_nl_clm: fsurdat = '/repopath/land/surfdata ... .nc' and flanduse_timeseries = '/repopath/land/landuse.timeseries ... .nc'. In your $REPO, create a directory 'land' (mkdir $REPO/land) and copy these 2 nc files to it.

• conda deactivate

[marcin-kupilas]

Set up CESM to run with grid

---

CCS_CONFIG

= /glade/derecho/scratch/mmkupilas/tag_cam6_4_028/ 1. Add an entry for your new ESMF mesh file in /ccs_config/component_grids_nuopc.xml ``` 79274 1 /glade/derecho/scratch/mmkupilas/RR/Grids/ne0np4.tromso01.ne30x8/grids/tromso01_np4_MESH.nc ne0np4.tromso01.ne30x8 is a Spectral Elem 1-deg grid with a 1/8 deg refined region over Scandanavia Test support only ``` 2. Add a grid alias entry in /ccs_config/modelgrid_aliases_nuopc.xml ``` ne0np4.tromso01.ne30x8 ne0np4.tromso01.ne30x8 ne0np4.tromso01.ne30x8 tx0.1v3 ``` # Set up case for testing on derecho CREATE NEWCASE /glade/derecho/scratch/mmkupilas/tag_cam6_4_028/cime/scripts/create_newcase --case /glade/derecho/scratch/mmkupilas/cesm_sims/cam6_4_028/f.e23.FWmaHIST.ne0np4.tromso01.ne30x8.test001 --compset FWmaHIST --res ne0tromso0130x8_ne0tromso0130x8_mt12 --run-unsupported --project P93300043 --pecount 512 CASEDIR /glade/derecho/scratch/mmkupilas/cesm_sims/cam6_4_028/f.e23.FWmaHIST.ne0np4.tromso01.ne30x8.test001 RUNDIR /glade/derecho/scratch/mmkupilas/cesm_dirs/f.e23.FWmaHIST.ne0np4.tromso01.ne30x8.test001/run/ ARCHIVE /glade/derecho/scratch/mmkupilas/archive/f.e23.FWmaHIST.ne0np4.tromso01.ne30x8.test001 --- ./xmlchange RUNDIR=/glade/derecho/scratch/mmkupilas/cesm_dirs/f.e23.FWmaHIST.ne0np4.tromso01.ne30x8.test001/run/ Change timestep based on recommended settings in https://github.com/ESMCI/Community_Mesh_Generation_Toolkit/blob/master/VRM_tools/Docs/CAM-tsteps-inic-for-newgrids_v0.pdf ./xmlchange ATM_NCPL=384 Add the override flag in your individual case directory to ensure CTSM doesn’t error out due to an unsupported grid: ./xmlchange --append CLM_BLDNML_OPTS="-no-chk_res" --- Add to user_nl_cam to be consistent with [f.e23.FWmaHIST.ne30pg3_me30pg3_mg17_L135.001](https://github.com/marcin-kupilas/MesoS2D-modelling/issues/1#top) ``` ncdata='/glade/derecho/scratch/mmkupilas/RR/Grids/ne0np4.tromso01.ne30x8/inic/ne0np4.tromso01.ne30x8_L135_1979-01-11_c240820.nc' bnd_topo='/glade/derecho/scratch/mmkupilas/RR/Grids/ne0np4.tromso01.ne30x8/topo/tromso01_gmted2010_modis_bedmachine_nc3000_Laplace0100_noleak_20241001.nc' drydep_srf_file='/glade/derecho/scratch/mmkupilas/RR/Grids/ne0np4.tromso01.ne30x8/atmsrf/atmsrf_ne0np4.tromso01.ne30x8_240820.nc' se_refined_mesh=.true. se_mesh_file='/glade/derecho/scratch/mmkupilas/RR/Grids/ne0np4.tromso01.ne30x8/grids/tromso01_EXODUS.nc' inithist = 'DAILY' interpolate_output = .false.,.true., interpolate_nlat = " ",192 interpolate_nlon = " ",288 empty_htapes=.true. fincl1 = 'U','V','Q','T','O','NO','O3','PSL','OMEGA','PS','PRECT','OMEGA500','OMEGA850' fincl2 = 'U','V','Q','T','O','NO','O3','PSL','OMEGA','PS','PRECT','OMEGA500','OMEGA850' mfilt = 1, 1 avgflag_pertape = 'I','I' nhtfrq = 1,1 phys_grid_ctem_nfreq = -6 ! MMK "nfreq" bewteen TEM calculations phys_grid_ctem_za_nlat = 90 phys_grid_ctem_zm_nbas = 120 ! Uncomment this line if you want to turn-off the PBL-based moving mountain GWs use_gw_movmtn_pbl = .false. qbo_use_forcing = .false. ! TRUE implies qbo package is active. use_gw_rdg_beta = .true. use_gw_convect_dp = .true. use_gw_front = .true. effgw_beres_dp = 0.70D0 ! Efficiency associated with convective gravity waves from the Beres scheme (deep convection). gw_qbo_hdepth_scaling = 0.25D0 ! Scaling factor for heating depth in gravity waves from convection. If less than 1.0 this acts as an effective reduction of the gravity wave phase speeds needed to drive the QBO. effgw_cm = 1.0D0 ! Efficiency associated with gravity waves from frontogenesis. effgw_rdg_beta = 1.2D0 effgw_rdg_beta_max = 1.2D0 gw_top_taper = .false. ! MMK Whether or not to apply tapering at the top of the model (above 0.6E-02 Pa) to reduce undesired effects of gravity waves in the thermosphere/ionosphere. ``` Add to user_nl_clm ``` fsurdat='/glade/derecho/scratch/mmkupilas/RR/Grids/ne0np4.tromso01.ne30x8/ctsm5.2.007/tools/mksurfdata_esmf/surfdata_tromso01.ne30x8_SSP3-7.0_1979_78pfts_c241001.nc' flanduse_timeseries='/glade/derecho/scratch/mmkupilas/RR/Grids/ne0np4.tromso01.ne30x8/ctsm5.2.007/tools/mksurfdata_esmf/landuse.timeseries_tromso01.ne30x8_SSP3-7.0_1979-2026_78pfts_c241001.nc' ``` --- XML changes ./xmlchange NTASKS=-20 ./xmlchange RUN_REFDATE=1980-01-01 ./xmlchange RUN_STARTDATE=1980-01-01 ./xmlchange STOP_OPTION=ndays ./xmlchange STOP_N=1 ./xmlchange CONTINUE_RUN=FALSE ./xmlchange CAM_CONFIG_OPTS="-phys cam7 -nlev 135 -age_of_air_trcs -chem waccm_ma_mam5 -cppdefs -Dwaccm_debug -pcols 9" # to be consistent with fiducial case ./case.setup --reset # Because of NTASKS change ./case.build ./case.submit WAITING ON FIRST RUN