[develop] Use uwtools instead of set_namelist

christinaholtNOAA commented 4 months ago

DESCRIPTION OF CHANGES:

Continues the integration of the uwtools package. In this PR, I've done the following:

Call the UW config tool instead of set_namelist using the uwtools CLI in bash scripts and API in Python scripts
Lint the ush/set_fv3nml*.py files
Update uwtools to the latest release version

Type of change

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

TESTS CONDUCTED:

[x] hera.intel
[ ] orion.intel
[ ] hercules.intel
[ ] cheyenne.intel
[ ] cheyenne.gnu
[ ] derecho.intel
[ ] gaea.intel
[ ] gaeac5.intel
[ ] jet.intel
[ ] wcoss2.intel
[ ] NOAA Cloud (indicate which platform)
[ ] Jenkins
[x] fundamental test suite
[ ] comprehensive tests (specify which if a subset was used)

The final test is still running (slowly) after updating to the latest version of uwtools. I don't expect trouble out of it give our changes between uwtools v2.0.1 and v2.1.0 were backwards compatible and this PR worked as expected on Hera with uwtools v.2.0.1.

DEPENDENCIES:

n/a

DOCUMENTATION:

I don't think so, but could have missed something.

ISSUE:

n/a

CHECKLIST

[x] My code follows the style guidelines in the Contributor's Guide
[x] I have performed a self-review of my own code using the Code Reviewer's Guide
[x] I have commented my code, particularly in hard-to-understand areas
[ ] My changes need updates to the documentation. I have made corresponding changes to the documentation
[x] My changes do not require updates to the documentation (explain).
[x] My changes generate no new warnings
[x] New and existing tests pass with my changes
[x] Any dependent changes have been merged and published

MichaelLueken commented 4 months ago

Manual testing of the WE2E coverage tests on Hera Intel have successfully completed:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
custom_ESGgrid_Peru_12km_20240318190205                            COMPLETE              18.31
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2019061200_2024031  COMPLETE               6.49
get_from_HPSS_ics_GDAS_lbcs_GDAS_fmt_netcdf_2022040400_ensemble_2  COMPLETE             762.25
get_from_HPSS_ics_HRRR_lbcs_RAP_20240318190209                     COMPLETE              14.46
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20240  COMPLETE               6.24
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20  COMPLETE              13.14
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_RAP_20240318190213  COMPLETE              10.44
grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2_20240  COMPLETE               6.48
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_202403  COMPLETE             232.17
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_20240318  COMPLETE             307.73
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_HRRR_202403181  COMPLETE             334.05
pregen_grid_orog_sfc_climo_20240318190219                          COMPLETE               8.16
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE            1719.92

MichaelLueken commented 3 months ago

@christinaholtNOAA -

The manual Hera GNU WE2E coverage tests successfully passed on Rocky8:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
custom_ESGgrid_Central_Asia_3km_20240318202335                     COMPLETE              37.79
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2019061200_202403  COMPLETE              11.19
get_from_NOMADS_ics_FV3GFS_lbcs_FV3GFS_20240318202337              COMPLETE              16.66
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_2024031820  COMPLETE              44.77
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_202  COMPLETE              25.55
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0_20240318202  COMPLETE              19.99
long_fcst_20240318202342                                           COMPLETE              72.13
MET_verification_only_vx_20240318202343                            COMPLETE               0.26
MET_ensemble_verification_only_vx_time_lag_20240318202345          COMPLETE               9.42
nco_grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_202  COMPLETE              60.01
2019_halloween_storm_20240318202350                                COMPLETE              52.46
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             350.23

The manual launched fundamental tests successfully passed on Orion:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              11.17
nco_grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_timeoffset_suite_  COMPLETE              13.59
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20240  COMPLETE               9.42
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              17.86
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024031  COMPLETE              33.18
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0_20240318134  COMPLETE              16.18
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2024031813460  COMPLETE              24.59
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             125.99

The automated Gaea and Hercules tests successfully passed.

On Jet, the grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2 test failed in run_fcst. The use of rocotorewind/rocotoboot allowed this test to successfully pass:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
community_20240318192628                                           COMPLETE              21.54
custom_ESGgrid_20240318192631                                      COMPLETE              19.49
custom_ESGgrid_Great_Lakes_snow_8km_20240318192632                 COMPLETE              17.65
custom_GFDLgrid_20240318192634                                     COMPLETE              14.12
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2021032018_202403  COMPLETE              13.02
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_netcdf_2022060112_48h_20  COMPLETE              59.30
get_from_HPSS_ics_RAP_lbcs_RAP_20240318192638                      COMPLETE              18.18
grid_RRFS_AK_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_20240318192640  COMPLETE             224.65
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20  COMPLETE              43.69
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20240  COMPLETE               7.65
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_2024  COMPLETE             505.07
nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR_2024  COMPLETE              10.96
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             955.32

On Derecho, the grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_HRRR test failed in run_fcst_mem001 and run_fcst_mem002. The logs for this test can be found - /glade/derecho/scratch/epicufsrt/jenkins/workspace/s-srweather-app_pipeline_PR-1054/derecho/expt_dirs/grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_HRRR/log. The traceback is:

Traceback (most recent call last):
  File "/glade/derecho/scratch/epicufsrt/jenkins/workspace/s-srweather-app_pipeline_PR-1054/derecho/ush/set_fv3nml_ens_stoch_seeds.py", line 145, in <module>
    set_fv3nml_ens_stoch_seeds(args.cdate, config)
  File "/glade/derecho/scratch/epicufsrt/jenkins/workspace/s-srweather-app_pipeline_PR-1054/derecho/ush/set_fv3nml_ens_stoch_seeds.py", line 48, in set_fv3nml_ens_stoch_seeds
    import_vars(dictionary=expt_config["global"])
                           ~~~~~~~~~~~^^^^^^^^^^
  File "/glade/derecho/scratch/epicufsrt/jenkins/workspace/s-srweather-app_pipeline_PR-1054/derecho/conda/envs/srw_app/lib/python3.11/collections/__init__.py", line 1126, in __getitem__
    raise KeyError(key)
KeyError: 'global'

I have also opened PR #3 in your fork. This PR will change the Hera spack-stack from CentOS to Rocky8.

Once PR #3 has been merged and the issue with grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_HRRR has been addressed on Derecho, I will move forward with merging this PR.

MichaelLueken commented 3 months ago

The Derecho WE2E coverage tests are now passing:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
custom_ESGgrid_IndianOcean_6km_20240319130911                      COMPLETE              28.47
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20  COMPLETE              46.09
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2024031913091  COMPLETE              53.24
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_HRRR_20240319  COMPLETE              35.42
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              21.59
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024031913092  COMPLETE              46.95
nco_grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_timeoffset_suite_  COMPLETE              27.61
pregen_grid_orog_sfc_climo_20240319130928                          COMPLETE              21.14
specify_template_filenames_20240319130931                          COMPLETE              21.72
2019_hurricane_barry_20240319130932                                COMPLETE              41.34
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             343.57

The Gaea WE2E coverage tests were also ran to ensure that the deterministic stochastic test still functions. All tests successfully passed:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
community_20240319161319                                           COMPLETE              48.59
custom_ESGgrid_NewZealand_3km_20240319161327                       COMPLETE              54.69
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              36.58
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP_20240319161  COMPLETE              40.97
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_2024031916  COMPLETE              39.66
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson  COMPLETE             360.78
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024031  COMPLETE              43.79
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_20  COMPLETE             297.49
grid_SUBCONUS_Ind_3km_ics_RAP_lbcs_RAP_suite_RRFS_v1beta_plot_202  COMPLETE              22.08
nco_ensemble_20240319161414                                        COMPLETE             121.49
nco_grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thom  COMPLETE             347.88
2020_CAPE_20240319161437                                           COMPLETE              43.40
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE            1457.40

Moving forward with merging this PR now.

ufs-community / ufs-srweather-app