[develop] Changes for Rocky8 on Hera

DESCRIPTION OF CHANGES:

Hera is switching from CentOS to Rocky OS. 1) One of changes needed for SRW to run on both Rocky and CentOS is to update cmake version from 3.20.1 to 3.23.1 2) Some HPSS batch jobs need more than 2GB 3) Fix in bash scripts/exregional_make_lbcs.sh

Hera comprehensive tests on CentOS passed. Hera comprehensive tests on Rocky passed.

Type of change

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

TESTS CONDUCTED:

[x] hera.intel
[ ] orion.intel
[ ] hercules.intel
[ ] cheyenne.intel
[ ] cheyenne.gnu
[ ] derecho.intel
[ ] gaea.intel
[ ] gaeac5.intel
[ ] jet.intel
[ ] wcoss2.intel
[ ] NOAA Cloud (indicate which platform)
[ ] Jenkins
[ ] fundamental test suite
[x] comprehensive tests

ISSUE:

Solves issue #1033

CHECKLIST

[x] My code follows the style guidelines in the Contributor's Guide
[x] I have performed a self-review of my own code using the Code Reviewer's Guide
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] My changes need updates to the documentation. I have made corresponding changes to the documentation
[x] My changes do not require updates to the documentation (explain).
[x] My changes generate no new warnings
[x] New and existing tests pass with my changes
[ ] Any dependent changes have been merged and published

LABELS (optional):

A Code Manager needs to add the following labels to this PR:

[ ] Work In Progress
[ ] bug
[x] enhancement
[ ] documentation
[ ] release
[ ] high priority
[ ] run_ci
[ ] run_we2e_fundamental_tests
[ ] run_we2e_comprehensive_tests
[ ] Needs Cheyenne test
[ ] Needs Jet test
[ ] Needs Hera test
[ ] Needs Orion test
[ ] help wanted

All 74 experiments finished
Calculating core-hour usage and printing final summary
----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
2020_CAD_20240217184749                                            COMPLETE              33.38
community_20240217184750                                           COMPLETE              19.02
custom_ESGgrid_20240217184751                                      COMPLETE              13.57
custom_ESGgrid_Central_Asia_3km_20240217184753                     COMPLETE              26.29
custom_ESGgrid_Great_Lakes_snow_8km_20240217184754                 COMPLETE              13.24
custom_ESGgrid_IndianOcean_6km_20240217184755                      COMPLETE              18.87
custom_ESGgrid_NewZealand_3km_20240217184756                       COMPLETE              53.55
custom_ESGgrid_Peru_12km_20240217184758                            COMPLETE              19.84
custom_ESGgrid_SF_1p1km_20240217184759                             COMPLETE             156.45
custom_GFDLgrid__GFDLgrid_USE_NUM_CELLS_IN_FILENAMES_eq_FALSE_202  COMPLETE               9.45
custom_GFDLgrid_20240217184801                                     COMPLETE               9.16
deactivate_tasks_20240217184802                                    COMPLETE               0.90
get_from_AWS_ics_GEFS_lbcs_GEFS_fmt_grib2_2022040400_ensemble_2me  COMPLETE             736.63
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2019061200_2024021  COMPLETE               7.71
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2019061200_202402  COMPLETE              10.13
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2021032018_202402  COMPLETE               9.57
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_netcdf_2022060112_48h_20  COMPLETE              59.15
get_from_HPSS_ics_GDAS_lbcs_GDAS_fmt_netcdf_2022040400_ensemble_2  COMPLETE             755.88
get_from_HPSS_ics_GSMGFS_lbcs_GSMGFS_20240217184811                COMPLETE               7.00
get_from_HPSS_ics_HRRR_lbcs_RAP_20240217184812                     COMPLETE              14.58
get_from_HPSS_ics_RAP_lbcs_RAP_20240217184814                      COMPLETE              17.67
get_from_NOMADS_ics_FV3GFS_lbcs_FV3GFS_20240217184815              COMPLETE              11.81
grid_CONUS_25km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_202  COMPLETE               9.47
grid_CONUS_3km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_  COMPLETE             259.46
grid_RRFS_AK_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20240  COMPLETE             136.19
grid_RRFS_AK_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_20240217184820  COMPLETE             174.68
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP_20240217184  COMPLETE              25.40
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20  COMPLETE              31.20
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_2024021718  COMPLETE              28.50
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_202  COMPLETE              28.49
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20240  COMPLETE               6.91
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20  COMPLETE              13.04
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              13.50
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_2024021718  COMPLETE              16.68
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP_20240217184  COMPLETE              43.40
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_202  COMPLETE              18.35
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_RAP_20240217184833  COMPLETE              10.64
grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2_20240  COMPLETE               7.13
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2024021718483  COMPLETE              21.82
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_RRFS_v1beta_202402171  COMPLETE              16.68
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_202402  COMPLETE             230.69
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson  COMPLETE             324.57
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_20240217  COMPLETE             304.34
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_20240217184  COMPLETE             368.72
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_2024  COMPLETE             365.46
grid_RRFS_CONUScompact_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_  COMPLETE              26.35
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_HRRR_20240217  COMPLETE              27.29
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              25.90
grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_  COMPLETE              10.21
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024021  COMPLETE              27.95
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE               9.47
grid_RRFS_CONUScompact_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_2  COMPLETE             274.23
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_HRRR_202402171  COMPLETE             325.11
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_20  COMPLETE             327.15
grid_RRFS_NA_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP_20240217184855  COMPLETE              70.67
grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_WoFS_v0_202402  COMPLETE              15.31
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024021718485  COMPLETE              19.00
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0_20240217184  COMPLETE              14.41
grid_SUBCONUS_Ind_3km_ics_NAM_lbcs_NAM_suite_GFS_v16_202402171849  COMPLETE              20.54
grid_SUBCONUS_Ind_3km_ics_RAP_lbcs_RAP_suite_RRFS_v1beta_plot_202  COMPLETE               9.42
long_fcst_20240217184903                                           COMPLETE              59.21
MET_ensemble_verification_only_vx_20240217184904                   COMPLETE               1.07
MET_ensemble_verification_only_vx_time_lag_20240217184907          COMPLETE               3.82
MET_ensemble_verification_winter_wx_20240217184910                 COMPLETE             118.84
MET_verification_only_vx_20240217184912                            COMPLETE               0.23
nco_20240217184915                                                 COMPLETE               8.46
nco_ensemble_20240217184917                                        COMPLETE              68.93
nco_grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_202  COMPLETE              31.96
nco_grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_timeoffset_suite_  COMPLETE              13.71
nco_grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thom  COMPLETE             329.10
nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR_2024  COMPLETE              11.68
pregen_grid_orog_sfc_climo_20240217184929                          COMPLETE               8.78
specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS_20240217184931               COMPLETE               7.20
specify_template_filenames_20240217184932                          COMPLETE               9.08
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE            6334.25

Detailed summary written to /scratch2/NCEPDEV/fv3-cam/Ratko.Vasic/CentOS/expt_dirs/WE2E_summary_20240218065224.txt

Finished all tests on Rocky OS:

All 74 experiments finished
Calculating core-hour usage and printing final summary
----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
2020_CAD_20240220234240                                            COMPLETE              34.47
community_20240220234241                                           COMPLETE              21.23
custom_ESGgrid_20240220234243                                      COMPLETE              14.78
custom_ESGgrid_Central_Asia_3km_20240220234244                     COMPLETE              26.25
custom_ESGgrid_Great_Lakes_snow_8km_20240220234245                 COMPLETE              12.40
custom_ESGgrid_IndianOcean_6km_20240220234247                      COMPLETE              20.84
custom_ESGgrid_NewZealand_3km_20240220234249                       COMPLETE              54.41
custom_ESGgrid_Peru_12km_20240220234252                            COMPLETE              18.40
custom_ESGgrid_SF_1p1km_20240220234255                             COMPLETE             161.37
custom_GFDLgrid__GFDLgrid_USE_NUM_CELLS_IN_FILENAMES_eq_FALSE_202  COMPLETE               7.70
custom_GFDLgrid_20240220234301                                     COMPLETE               7.37
deactivate_tasks_20240220234303                                    COMPLETE               0.86
get_from_AWS_ics_GEFS_lbcs_GEFS_fmt_grib2_2022040400_ensemble_2me  COMPLETE             754.56
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2019061200_2024022  COMPLETE               6.32
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2019061200_202402  COMPLETE               8.84
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2021032018_202402  COMPLETE               9.04
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_netcdf_2022060112_48h_20  COMPLETE              63.43
get_from_HPSS_ics_GDAS_lbcs_GDAS_fmt_netcdf_2022040400_ensemble_2  COMPLETE             774.93
get_from_HPSS_ics_GSMGFS_lbcs_GSMGFS_20240220234317                COMPLETE               6.59
get_from_HPSS_ics_HRRR_lbcs_RAP_20240220234319                     COMPLETE              16.67
get_from_HPSS_ics_RAP_lbcs_RAP_20240220234320                      COMPLETE              18.51
get_from_NOMADS_ics_FV3GFS_lbcs_FV3GFS_20240220234322              COMPLETE              11.81
grid_CONUS_25km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_202  COMPLETE               8.41
grid_CONUS_3km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_  COMPLETE             263.14
grid_RRFS_AK_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20240  COMPLETE             140.47
grid_RRFS_AK_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_20240220234329  COMPLETE             180.51
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP_20240220234  COMPLETE              24.72
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20  COMPLETE              31.63
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_2024022023  COMPLETE              29.06
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_202  COMPLETE              29.08
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20240  COMPLETE               6.21
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20  COMPLETE              13.52
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              14.05
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_2024022023  COMPLETE              15.81
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP_20240220234  COMPLETE              41.06
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_202  COMPLETE              19.27
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_RAP_20240220234354  COMPLETE              10.67
grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2_20240  COMPLETE               6.74
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2024022023435  COMPLETE              22.86
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_RRFS_v1beta_202402202  COMPLETE              16.96
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_202402  COMPLETE             234.56
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson  COMPLETE             334.56
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_20240220  COMPLETE             314.63
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_20240220234  COMPLETE             378.31
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_2024  COMPLETE             371.23
grid_RRFS_CONUScompact_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_  COMPLETE              27.26
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_HRRR_20240220  COMPLETE              27.38
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              25.81
grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_  COMPLETE               9.63
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024022  COMPLETE              32.22
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE               9.77
grid_RRFS_CONUScompact_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_2  COMPLETE             281.43
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_HRRR_202402202  COMPLETE             332.71
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_20  COMPLETE             333.16
grid_RRFS_NA_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP_20240220234442  COMPLETE              69.65
grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_WoFS_v0_202402  COMPLETE              18.08
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024022023444  COMPLETE              17.09
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0_20240220234  COMPLETE              16.05
grid_SUBCONUS_Ind_3km_ics_NAM_lbcs_NAM_suite_GFS_v16_202402202344  COMPLETE              20.90
grid_SUBCONUS_Ind_3km_ics_RAP_lbcs_RAP_suite_RRFS_v1beta_plot_202  COMPLETE               8.58
long_fcst_20240220234503                                           COMPLETE              54.55
MET_ensemble_verification_only_vx_20240220234506                   COMPLETE               1.00
MET_ensemble_verification_only_vx_time_lag_20240220234513          COMPLETE               3.64
MET_ensemble_verification_winter_wx_20240220234519                 COMPLETE             120.46
MET_verification_only_vx_20240220234523                            COMPLETE               0.23
nco_20240220234526                                                 COMPLETE               8.45
nco_ensemble_20240220234528                                        COMPLETE              62.75
nco_grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_202  COMPLETE              27.68
nco_grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_timeoffset_suite_  COMPLETE              11.75
nco_grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thom  COMPLETE             324.59
nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR_2024  COMPLETE              11.87
pregen_grid_orog_sfc_climo_20240220234545                          COMPLETE               6.60
specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS_20240220234549               COMPLETE               6.38
specify_template_filenames_20240220234550                          COMPLETE               8.35
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE            6436.26

Detailed summary written to /scratch2/NCEPDEV/fv3-cam/Ratko.Vasic/Rocky8/expt_dirs/WE2E_summary_20240221021138.txt

The Rocky8 fundamental tests have also successfully passed:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE               9.55
nco_grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_timeoffset_suite_  COMPLETE              12.49
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20240  COMPLETE               7.75
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              14.78
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024022  COMPLETE              31.61
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0_20240222153  COMPLETE              15.16
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2024022215314  COMPLETE              22.06
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             113.40

@RatkoVasic-NOAA -

Before launching the automated Jenkins tests, since the xJet partition can only be accessed with Rocky8, please modify your Rocky-pt1 branch's ush/machine/jet.yaml file so that PARTITION_FCST uses vJet instead of xJet. Once complete, I will add the Jenkins label. Thanks!

While requeued in Jenkins, there was a single failure on Orion's coverage WE2E test. I was able to successfully run the coverage tests manually on Orion:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
custom_ESGgrid_SF_1p1km_20240223133216                             COMPLETE             162.16
deactivate_tasks_20240223133217                                    COMPLETE               1.49
get_from_AWS_ics_GEFS_lbcs_GEFS_fmt_grib2_2022040400_ensemble_2me  COMPLETE             761.21
grid_CONUS_3km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_  COMPLETE             253.80
grid_RRFS_AK_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20240  COMPLETE             140.04
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_RRFS_v1beta_202402231  COMPLETE              14.86
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_20240223133  COMPLETE             375.60
grid_RRFS_CONUScompact_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_  COMPLETE              30.13
grid_RRFS_CONUScompact_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_2  COMPLETE             278.05
grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_WoFS_v0_202402  COMPLETE              14.27
nco_20240223133229                                                 COMPLETE               8.30
2020_CAD_20240223133231                                            COMPLETE              33.48
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE            2073.39

Hopefully, the retest will pass as well.

The Derecho pipeline has been requeued, but the manual WE2E coverage tests have successfully passed on the machine:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
custom_ESGgrid_IndianOcean_6km_20240223113012                      COMPLETE              31.55
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20  COMPLETE              49.28
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2024022311301  COMPLETE              55.56
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_HRRR_20240223  COMPLETE              37.97
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              22.62
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024022311302  COMPLETE              50.44
nco_grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_timeoffset_suite_  COMPLETE              30.27
pregen_grid_orog_sfc_climo_20240223113030                          COMPLETE              19.65
specify_template_filenames_20240223113032                          COMPLETE              21.18
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             318.52

Hopefully the rerun will also pass.

The Jenkins tests successfully passed on Hera GNU:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
custom_ESGgrid_Central_Asia_3km_20240224092412                     COMPLETE              36.72
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2019061200_202402  COMPLETE              11.18
get_from_NOMADS_ics_FV3GFS_lbcs_FV3GFS_20240224092415              COMPLETE              18.35
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_2024022409  COMPLETE              44.57
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_202  COMPLETE              27.31
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0_20240224092  COMPLETE              20.11
long_fcst_20240224092419                                           COMPLETE              68.26
MET_verification_only_vx_20240224092420                            COMPLETE               0.24
MET_ensemble_verification_only_vx_time_lag_20240224092423          COMPLETE               9.24
nco_grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_202  COMPLETE              61.56
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             297.54

and Hera Intel:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
custom_ESGgrid_Peru_12km_20240224092356                            COMPLETE              17.03
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2019061200_2024022  COMPLETE               5.99
get_from_HPSS_ics_GDAS_lbcs_GDAS_fmt_netcdf_2022040400_ensemble_2  COMPLETE             770.37
get_from_HPSS_ics_HRRR_lbcs_RAP_20240224092359                     COMPLETE              14.07
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20240  COMPLETE               5.97
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20  COMPLETE              12.60
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_RAP_20240224092404  COMPLETE               9.81
grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2_20240  COMPLETE               6.78
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_202402  COMPLETE             233.25
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_20240224  COMPLETE             300.92
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_HRRR_202402240  COMPLETE             323.22
pregen_grid_orog_sfc_climo_20240224092410                          COMPLETE               8.01
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE            1708.02

The Jenkins retest on Orion successfully passed:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
custom_ESGgrid_SF_1p1km_20240224084700                             COMPLETE             166.51
deactivate_tasks_20240224084702                                    COMPLETE               1.30
get_from_AWS_ics_GEFS_lbcs_GEFS_fmt_grib2_2022040400_ensemble_2me  COMPLETE             750.79
grid_CONUS_3km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_  COMPLETE             253.05
grid_RRFS_AK_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20240  COMPLETE             143.07
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_RRFS_v1beta_202402240  COMPLETE              16.39
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_20240224084  COMPLETE             382.33
grid_RRFS_CONUScompact_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_  COMPLETE              31.49
grid_RRFS_CONUScompact_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_2  COMPLETE             272.44
grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_WoFS_v0_202402  COMPLETE              14.92
nco_20240224084714                                                 COMPLETE               8.61
2020_CAD_20240224084717                                            COMPLETE              33.39
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE            2074.29

The Jenkins retest on Derecho successfully passed:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
custom_ESGgrid_IndianOcean_6km_20240224101930                      COMPLETE              29.05
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20  COMPLETE              45.54
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2024022410194  COMPLETE              53.38
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_HRRR_20240224  COMPLETE              37.19
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              21.48
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024022410194  COMPLETE              47.53
nco_grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_timeoffset_suite_  COMPLETE              29.96
pregen_grid_orog_sfc_climo_20240224101951                          COMPLETE              20.78
specify_template_filenames_20240224101954                          COMPLETE              20.91
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             305.82

Hung jobs on Jet (trying to use xJet while using CentOS login nodes) kept the Jet tests from moving forward. I canceled these hung jobs in the role.epic space and the final testing phase has been launched. Once complete, I will be able to merge this PR.

The Jenkins tests on Jet have successfully passed:

---------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
community_20240226151727                                           COMPLETE              18.41
custom_ESGgrid_20240226151731                                      COMPLETE              16.52
custom_ESGgrid_Great_Lakes_snow_8km_20240226151732                 COMPLETE              13.54
custom_GFDLgrid_20240226151735                                     COMPLETE               9.22
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2021032018_202402  COMPLETE               9.67
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_netcdf_2022060112_48h_20  COMPLETE              50.80
get_from_HPSS_ics_RAP_lbcs_RAP_20240226151748                      COMPLETE              15.06
grid_RRFS_AK_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_20240226151752  COMPLETE             212.71
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20  COMPLETE              39.62
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20240  COMPLETE               7.64
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_2024  COMPLETE             484.59
nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR_2024  COMPLETE              10.27
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             888.05

Moving forward with merging this PR.

ufs-community / ufs-srweather-app