Use ESMF 8.2.0 library - Githubissues

MinsukJi-NOAA commented 2 years ago

Update the ESMF library from 8.1.0 to 8.2.0. This change is needed to utilize latest fixes in the ESMF; see, for example, #143.

A corresponding PR will be made, in which regression (consistency) tests for those utilities affected by the ESMF library (chgres_cube and grid_gen) on all 5 platforms will be run and comparison of results be posted.

Part of #143.

MinsukJi-NOAA commented 2 years ago

@GeorgeGayno-NOAA These are some of the differences from using ESMF 8.2.0. There are LOTS of these statistics to consider since there are many failed consistency tests on all 5 platforms. Wanted to check with you if you have any suggestions on how to go about analyzing these systematically.

+ nccmp -dmfqS out.atm.tile1.nc /scratch1/NCEPDEV/nems/role.ufsutils/ufs_utils/reg_tests/chgres_cube/baseline_data/c96_fv3_restart/out.atm.tile1.nc
Variable Group Count          Sum      AbsSum          Min         Max       Range         Mean      StdDev
u_w      /         2 -2.23517e-07  2.5332e-07 -2.38419e-07 1.49012e-08  2.5332e-07 -1.11759e-07 1.79124e-07
v_w      /        16  8.19564e-08 1.37836e-06 -2.38419e-07 4.76837e-07 7.15256e-07  5.12227e-09 1.56207e-07

+ nccmp -dmfqS out.atm.tile7.nc /lfs4/HFIP/hfv3gfs/emc.nemspara/role.ufsutils/ufs_utils/reg_tests/chgres_cube/baseline_data/3km_conus_hrrr_gfssdf_grib2/out.atm.tile7.nc
Variable Group Count      Sum  AbsSum       Min      Max    Range        Mean    StdDev
u_w      /      1152  1.15363 50.6469 -0.190329 0.216066 0.406396  0.00100141 0.0529021
v_w      /      1152  1.18029 27.8806 -0.379808 0.261978 0.641786  0.00102456 0.0378907
u_s      /      1088 -20.8364 37.2089 -0.177918 0.108623 0.286541  -0.0191511 0.0372238
v_s      /      1088 -6.41019 33.7436 -0.127974 0.139483 0.267457 -0.00589172 0.0372765

GeorgeGayno-NOAA commented 2 years ago

@GeorgeGayno-NOAA These are some of the differences from using ESMF 8.2.0. There are LOTS of these statistics to consider since there are many failed consistency tests on all 5 platforms. Wanted to check with you if you have any suggestions on how to go about analyzing these systematically.

+ nccmp -dmfqS out.atm.tile1.nc /scratch1/NCEPDEV/nems/role.ufsutils/ufs_utils/reg_tests/chgres_cube/baseline_data/c96_fv3_restart/out.atm.tile1.nc
Variable Group Count          Sum      AbsSum          Min         Max       Range         Mean      StdDev
u_w      /         2 -2.23517e-07  2.5332e-07 -2.38419e-07 1.49012e-08  2.5332e-07 -1.11759e-07 1.79124e-07
v_w      /        16  8.19564e-08 1.37836e-06 -2.38419e-07 4.76837e-07 7.15256e-07  5.12227e-09 1.56207e-07

+ nccmp -dmfqS out.atm.tile7.nc /lfs4/HFIP/hfv3gfs/emc.nemspara/role.ufsutils/ufs_utils/reg_tests/chgres_cube/baseline_data/3km_conus_hrrr_gfssdf_grib2/out.atm.tile7.nc
Variable Group Count      Sum  AbsSum       Min      Max    Range        Mean    StdDev
u_w      /      1152  1.15363 50.6469 -0.190329 0.216066 0.406396  0.00100141 0.0529021
v_w      /      1152  1.18029 27.8806 -0.379808 0.261978 0.641786  0.00102456 0.0378907
u_s      /      1088 -20.8364 37.2089 -0.177918 0.108623 0.286541  -0.0191511 0.0372238
v_s      /      1088 -6.41019 33.7436 -0.127974 0.139483 0.267457 -0.00589172 0.0372765

@MinsukJi-NOAA The differences from the first test look small. But the differences from the second test seem bigger than I would expect. I would visualize the cases with the big differences and make some difference plots with the baseline data. Also, the grid generation procedure depends on ESMF. So you will need to run the 'grid_gen' tests as well.

MinsukJi-NOAA commented 2 years ago

On Hera:

grid_gen tests all passed.
For chgres_cube tests, some of the failed cases with large-value differences are shown below. Left: ESMF 8.2.0; Middle: Difference; Right: ESMF 8.1.0.
- c192_fv3_history

02_c192_fv3_history_out_atm_tile3_v_s

c192_gfs_grib2

08_c192_gfs_grib2_out_atm_tile3_u_w

3km_conus_hrrr_gfsdf_grib2

10_3km_conus_hrrr_gfsdf_grib2_out_atm_tile7_u_s

3km_conus_hrrr_newsfc_grib2

11_3km_conus_hrrr_newsfc_grib2_out_atm_tile7_u_s

GeorgeGayno-NOAA commented 2 years ago

@MinsukJi-NOAA Thanks for making these plots. I was looking at the ESMF 8.2.0 release notes:

https://github.com/esmf-org/esmf/releases/tag/ESMF_8_2_0

There were some changes to the regridding which result in small differences in the regridding weights. So that probably explains what we are seeing. I may look at the C192 case more closely myself. But I am inclined to make the upgrade to 8.2.0.

GeorgeGayno-NOAA commented 2 years ago

@MinsukJi-NOAA Where are your test data? I want to take a look.

GeorgeGayno-NOAA commented 2 years ago

@MinsukJi-NOAA What branch are you working from?

MinsukJi-NOAA commented 2 years ago

@GeorgeGayno-NOAA I am working from the latest develop branch: 04700f9 My test data on Hera is here: /scratch2/NCEPDEV/stmp1/Minsuk.Ji/reg-tests/chgres-cube Summary log is located here: /scratch1/NCEPDEV/stmp4/Minsuk.Ji/UFS_UTILS_04700f9/reg_tests/chgres_cube

GeorgeGayno-NOAA commented 2 years ago

@GeorgeGayno-NOAA I am working from the latest develop branch: 04700f9 My test data on Hera is here: /scratch2/NCEPDEV/stmp1/Minsuk.Ji/reg-tests/chgres-cube Summary log is located here: /scratch1/NCEPDEV/stmp4/Minsuk.Ji/UFS_UTILS_04700f9/reg_tests/chgres_cube

@MinsukJi-NOAA Please create a branch off 'develop' in your fork for any work under this issue. See our wiki page for more details: https://github.com/ufs-community/UFS_UTILS/wiki/3.-Checking-out-code-and-making-changes

MinsukJi-NOAA commented 2 years ago

@GeorgeGayno-NOAA I am working from the latest develop branch: 04700f9 My test data on Hera is here: /scratch2/NCEPDEV/stmp1/Minsuk.Ji/reg-tests/chgres-cube Summary log is located here: /scratch1/NCEPDEV/stmp4/Minsuk.Ji/UFS_UTILS_04700f9/reg_tests/chgres_cube

@MinsukJi-NOAA Please create a branch off 'develop' in your fork for any work under this issue. See our wiki page for more details: https://github.com/ufs-community/UFS_UTILS/wiki/3.-Checking-out-code-and-making-changes

@GeorgeGayno-NOAA my working branch is here: https://github.com/MinsukJi-NOAA/UFS_UTILS/tree/feature/esmf_8_2_0. So far, I have run regression tests on hera, jet, and orion. I will also report results from jet and orion.

MinsukJi-NOAA commented 2 years ago

On Jet:

grid_gen tests all passed
For chgres_cube tests, some of the failed cases with large-value differences are shown below. Left: ESMF 8.2.0; Middle: Difference; Right: ESMF 8.1.0.
- c192_fv3_history

02_c192_fv3_history_out_atm_tile3_v_s

c192_gfs_grib2

08_c192_gfs_grib2_out_atm_tile5_u_w

3km_conus_hrrr_gfsdf_grib2

10_3km_conus_hrrr_gfsdf_grib2_out_atm_tile7_u_s

3km_conus_hrrr_newsfc_grib2

11_3km_conus_hrrr_newsfc_grib2_out_atm_tile7_u_s

MinsukJi-NOAA commented 2 years ago

On Orion:

grid_gen tests all passed
For chgres_cube tests, some of the failed cases with large-value differences are shown below. Left: ESMF 8.2.0; Middle: Difference; Right: ESMF 8.1.0.
- c192_fv3_history

02_c192_fv3_history_out_atm_tile3_u_s

c192_gfs_grib2

07_c192_gfs_grib2_out_atm_tile5_u_w

3km_conus_hrrr_gfsdf_grib2 and 3km_conus_hrrr_newsfc_grib2 have the usual high-value difference points in the center of tile7 (similar to those shown above for Hera and Jet)

GeorgeGayno-NOAA commented 2 years ago

To be complete, the workflow files should be updated to use ESMF v8.2.0:

https://github.com/ufs-community/UFS_UTILS/tree/develop/.github/workflows

@MinsukJi-NOAA I can help with that.

MinsukJi-NOAA commented 2 years ago

To be complete, the workflow files should be updated to use ESMF v8.2.0:

https://github.com/ufs-community/UFS_UTILS/tree/develop/.github/workflows

@MinsukJi-NOAA I can help with that.

@GeorgeGayno-NOAA thanks. I will look into it.

MinsukJi-NOAA commented 2 years ago

On WCOSS Cray:

All grid_gen tests failed. But an examination of statistics in consistency.log indicates the differences are ~ 10^-7.
All chgres_cube tests failed. Mostly, the differences are small. Case with significant differences are plotted below.
- c192_fv3_history

08_c192_fv3_history_out_atm_tile5_u_w

c192_gfs_grib2

14_c192_gfs_grib2_out_atm_tile5_u_w

3km_conus_hrrr_gfsdf_grib2 and 3km_conus_hrrr_newsfc_grib2 have the usual high-value difference points in the center of tile7 (similar to those shown above for Hera and Jet)

MinsukJi-NOAA commented 2 years ago

On WCOSS Dell:

All grid_gen tests passed
For chgres_cube tests, some of the failed cases with large-value differences are shown below. Left: ESMF 8.2.0; Middle: Difference; Right: ESMF 8.1.0.
- c192_fv3_history
- c192_gfs_gribs
- 3km_conus_hrrr_gfssdf_grib2 and 3km_conus_hrrr_newsfc_grib2 have the usual high-value difference points in the center of tile7 (similar to those shown above for Hera and Jet)

MinsukJi-NOAA commented 2 years ago

@GeorgeGayno-NOAA please note that a few other minor module changes had to be made for wcoss dell: https://github.com/MinsukJi-NOAA/UFS_UTILS/commit/12fe8a038a298518e37db8bb5b2bf95cadb67058 I am not sure if the g2 library version change from 3.4.1 to 3.4.2 requires testing ufs utilities other than chgres_cube and grid_gen

GeorgeGayno-NOAA commented 2 years ago

@MinsukJi-NOAA The differences from the baseline are small enough that I will not stop the merge.

ufs-community / UFS_UTILS

Use ESMF 8.2.0 library #621