chunhuazhou commented 6 months ago

The goal is to create a CONUS domain with 12km resolution and to prepare for building the RRFSv2 test system.

The workflow follows the NSSL MPAS run, using the GSL-NSSL MPAS model that has RUC-LSM and some other changes. I copied / linked the real-time workflow from /misc/whome/role.wrfruc/HRRRv5 and made some changes to run it for the CONUS-12km domain, initialized from GFS/ GEFS at /lfs4/BMC/wrfruc/Chunhua.Zhou/JEDI-tests/HRRRv5-12km.

Under /lfs4/BMC/wrfruc/Chunhua.Zhou/JEDI-tests/HRRRv5-12km, use workflow.ens/hrrrv5.xml to run GEFS initialized ensemble forecast and workflow/hrrrv5.xml to run GFS initialized deterministic forecast.

Step 1: Download global 12km uniform meshes from https://mpas-dev.github.io/

$ wget https://www2.mmm.ucar.edu/projects/mpas/atmosphere_meshes/x1.4096002.tar.gz

$ wget https://www2.mmm.ucar.edu/projects/mpas/atmosphere_meshes/x1.4096002_static.tar.gz

Untar and get the following files:

x1.4096002.graph.info

x1.4096002.cvt.part.256

x1.4096002.grid.nc

x1.4096002.static.nc

Step 2: Create a CONUS 12km domain from the global 12km meshes - grid.nc

Using the file conus.custom.pts from MPAS-Limited-Area/docs/points-examples/ and modify it for the desired conus domain:

name: conus12km_mpas
Type: custom
Point: 40.0, -100.0
50.0, -145.0
50.0, -50.0
20.0, -65.0
20.0, -129.0

Use the`create_region` python script from MPAS-Limited-Area/ to get grid.nc file for the customized CONUS domain:

$ create_region conus.custom.pts x1.4096002.grid.nc

Output: conus12km_mpas.grid.nc conus12km_mpas.graph.info

Step 3: Run init_atmosphere_model and get conus12km_mpas.static.nc

$ cp /lfs4/BMC/wrfruc/Chunhua.Zhou/JEDI-tests/HRRRv5-12km/exec/init_atmosphere_model . $ cp /lfs4/BMC/wrfruc/Chunhua.Zhou/JEDI-tests/skylab/mpas/MPAS-Model.master/namelist.init_atmosphere . $ cp /lfs4/BMC/wrfruc/Chunhua.Zhou/JEDI-tests/skylab/mpas/MPAS-Model.master/streams.init_atmosphere .

Modify streams.init_atmosphere:

<streams>
<immutable_stream name="input"
                  type="input"
                  filename_template="conus12km_mpas.grid.nc">
                  input_interval="initial_only" />
<immutable_stream name="output"
                  type="output"
                  filename_template="conus12km_mpas.static.nc"
                  packages="initial_conds"
                  output_interval="initial_only" />
<immutable_stream name="surface"
                  type="output"
                  filename_template="x1.40962.sfc_update.nc"
                  filename_interval="none"
                  packages="sfc_update"
                  output_interval="86400" />
<immutable_stream name="lbc"
                  type="output"
                  filename_template="lbc.$Y-$M-$D_$h.$m.$s.nc"
                  filename_interval="output_interval"
                  packages="lbcs"
                  output_interval="3:00:00" />
</streams>

Modify namelist.init_atmosphere:

&nhyd_model
    config_init_case = 7
    config_start_time = '2010-10-23_00:00:00'
    config_stop_time = '2010-10-23_00:00:00'
    config_theta_adv_order = 3
    config_coef_3rd_order = 0.25
    config_interface_projection = 'linear_interpolation'
/
&dimensions
    config_nvertlevels = 55
    config_nsoillevels = 4
    config_nfglevels = 38
    config_nfgsoillevels = 4
/
&data_sources
    config_geog_data_path = '/lfs4/NAGAPE/hpc-wof1/ywang/MPAS/WPS_GEOG/'
    config_met_prefix = 'CFSR'
    config_sfc_prefix = 'SST'
    config_fg_interval = 86400
    config_landuse_data = 'MODIFIED_IGBP_MODIS_NOAH'
    config_topo_data = 'GMTED2010'
    config_vegfrac_data = 'MODIS'
    config_albedo_data = 'MODIS'
    config_maxsnowalbedo_data = 'MODIS'
    config_supersample_factor = 3
    config_30s_supersample_factor = 1
    config_use_spechumd = false
/
&vertical_grid
    config_ztop = 30000.0
    config_nsmterrain = 1
    config_smooth_surfaces = true
    config_dzmin = 0.3
    config_nsm = 30
    config_tc_vertical_grid = true
    config_blend_bdy_terrain = false
/
&interpolation_control
    config_extrap_airtemp = 'lapse-rate'
/
&preproc_stages
    config_static_interp = true
    config_native_gwd_static = true
    config_vertical_grid = false
    config_met_interp = false
    config_input_sst = false
    config_frac_seaice = false
/
&io
    config_pio_num_iotasks = 0
    config_pio_stride = 1
/
&decomposition
    config_block_decomp_file_prefix = 'conus12km_mpas.graph.info.part.'

Run init_atmosphere_model, using the following example run script (run_init.sh):

#!/bin/sh
#SBATCH --account=nrtrr
##SBATCH --qos=debug
#SBATCH --nodes=1-1
#SBATCH --tasks-per-node=1
#SBATCH -t 00:55:00
#SBATCH --job-name=init_atmosphere
#SBATCH -o out.init_atmosphere

source /lfs4/BMC/wrfruc/Chunhua.Zhou/JEDI-tests/HRRRv5-12km/module_files/modulefile.jet
ulimit -s unlimited
srun init_atmosphere_model

Output: conus12km_mpas.static.nc

Step 4: Use metis to partition the mesh for parallel run

Copy the source code from MPAS-DATA/metis-5.1.0.tar.gz Under /lfs4/BMC/wrfruc/Chunhua.Zhou/JEDI-tests/skylab/mpas/metis-5.1.0

$ module load gnu/13.2.0 intel/2023.2.0 $ make config $ make

Example command to partition the mesh for using 120 cores: $ gpmetis -minconn -contig -niter=200 conus12km_mpas.graph.info 120

Output: conus12km_mpas.graph.info.part.120

Step 5: Run a deterministic MPAS forecast, using workflow/hrrrv5.xml , with GFS GRIB2 data as IC/LBC

Main namelist entries for running the 12-km CONUS test:

&nhyd_model
    config_time_integration_order = 2
    config_dt = 60.0
    config_number_of_sub_steps = 3
    config_dynamics_split_steps = 3
    config_len_disp                 = 12000.0

Suggested number of cores: 120 cores as tested on Jet.

Step 6: Run 30-member ensemble forecast, using workflow.ens/hrrrv5.xml, with GEFS GRIB2 data as IC/LBC

The 1-hour forecast files (deterministic and 30-member ensemble, initialized at 12Z 05/20/2024, valid at 13Z), along with the RAP PrepBUFR observations:

Jet: /lfs4/BMC/wrfruc/Chunhua.Zhou/MPAS-DATA/conus12km_mpas/2024052013

Hera: /scratch2/BMC/wrfruc/chunhua/data/conus12km_mpas/2024052013