[develop] Fixing several issues, including 966 (bash octal issue); add new winter weather verification test with staged data

[mkavulich]

DESCRIPTION OF CHANGES:

This started as a fairly simple change (don't they always) to fix Issue #966 and add a new winter weather verification test in support of the RRFS agile framework project and has since snowballed into solving a number of outstanding and newly discovered issues.

New test

• The new test MET_ensemble_verification_winter_wx is added. This test will exercise a number of yet-untested capabilities in the workflow, including a 10-member ensemble, snowfall verification with staged data (so can be run on all platforms, not just Jet and Hera), and several SPP settings.
• As part of this new test, snowfall observations will now be staged on all tier-1 platforms, as well as netCDF GFS data and other observation types, all for the date 2022020300

Resolved issues

• 966 Resolved: In several locations, an explicit conversion is done to ENSMEM_INDX to ensure it is a base-10 integer, to avoid problems with bash interpreting numbers with a leading zero as octal.

• 978 resolved: per discussion in a recent SRW code management meeting, give EXPT_SUBDIR a default value "experiment" to avoid unnecessary complications and work for users. Additionally, the default behavior if an experiment directory already exists is changed to "quit" rather than "delete"

• Issue mentioned in this discussion; the setting fhzero=6 is removed from the weather model namelist for CCPP suite FV3_GFS_v17_p8, which allows precipitation and other accumulations to be made every hour rather than 6 hours (SRW output is always hourly, so this makes sense). Also, update diag_table.FV3_GFS_v17_p8 so that all output files will be hourly
• Per discussion in #939, remove an unnecessary special case in get_crontab_contents.py for Derecho

Other fixes

• Some old files for unsupported/removed CCPP suites are removed
• Add some missing task dependencies for retrieving verification obs

General improvements

• Many improvements to verification obs-pulling task
  • NDAS observations are now retrieved for forecast hour zero, and a better obs file is retrieved for major obs times (00z, 06z, 12z, 18z) per EMC guidance
  • Better in-line comments/documentation
  • Standardize order and messaging for file-on-disk checks across all observation types
• Added explanatory comments for reflectivity field in diag_table files
• Update diag_table.FV3_GFS_v17_p8 so that all output files will be hourly
• Simplify task dependencies that rely on staged verification observations; these "get_obs" tasks should always be run (they check that the data exists before trying to retrieve it), so no need to make the dependency conditional
• Add a check in monitor_jobs.py to ensure the yaml file does not contain duplicate experiment directories
• Make sure the key in the experiment dictionary used by is unique by appending the current date/time to the exptdir name; additionally, set this key as the WORKFLOW_ID variable (so that it could be used in the workflow if necessary).

Type of change

• [x] Bug fix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [x] This change requires a documentation update

TESTS CONDUCTED:

Ran fundamental tests on several platforms. Ran some manual tests for the above-mentioned WE2E and monitor_jobs features, including catting together several WE2E yaml files in a way that previously failed to ensure they now succeed.

Ran all verification tests (including new test) on Hera, Orion, and Derecho to ensure nothing changed unexpectedly. I also compared existing tests to develop on Hera, and confirmed that the additional NDAS obs are now being pulled and used for verification tasks, which resulted in expected differences in verification results.

• [x] hera.intel
  • Fundamental tests
  • All verification tests
• [x] orion.intel
  • Fundamental tests
  • All verification tests (except MET_verification_winter_wx and MET_ensemble_verification_only_vx_time_lag, which only work on Jet and Hera)
• [ ] hercules.intel
• [ ] cheyenne.intel
• [ ] cheyenne.gnu
• [x] derecho.intel
  • Fundamental tests
  • All verification tests (except MET_verification_winter_wx and MET_ensemble_verification_only_vx_time_lag, which only work on Jet and Hera)
• [ ] gaea.intel
• [ ] gaeac5.intel
• [ ] jet.intel
• [ ] wcoss2.intel
• [ ] NOAA Cloud (indicate which platform)
• [ ] Jenkins
• [x] fundamental test suite
• [ ] comprehensive tests

DEPENDENCIES:

None

DOCUMENTATION:

Working on updates to documentation (mostly for the new test); will merge when ready

ISSUE:

• Resolves #966
• Resolves #978
• Resolves issue from discussion: https://github.com/orgs/ufs-community/discussions/33

CHECKLIST

• [x] My code follows the style guidelines in the Contributor's Guide
• [x] I have performed a self-review of my own code using the Code Reviewer's Guide
• [x] I have commented my code, particularly in hard-to-understand areas
• [ ] My changes need updates to the documentation. I have made corresponding changes to the documentation
• [x] My changes generate no new warnings
• [x] New and existing tests pass with my changes
• [x] Any dependent changes have been merged and published

CONTRIBUTORS:

Thanks to @EdwardSnyder-NOAA for help staging the new test data

[MichaelLueken]

The coverage WE2E tests were manually ran on Derecho and they all successfully passed:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
custom_ESGgrid_IndianOcean_6km_20240104115217                      COMPLETE              24.66
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20  COMPLETE              45.55
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2024010411521  COMPLETE              45.90
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_HRRR_20240104  COMPLETE              29.86
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              18.61
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024010411522  COMPLETE              41.38
nco_grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_timeoffset_suite_  COMPLETE              24.19
pregen_grid_orog_sfc_climo_20240104115227                          COMPLETE              22.15
specify_template_filenames_20240104115229                          COMPLETE              22.77
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             275.07

[MichaelLueken]

@mkavulich -

The MET_verification_only_vx WE2E test on Hercules failed in run_MET_PcpCombine_obs_APCP01h with the following message:

/work/noaa/epic/role-epic/jenkins/workspace/fs-srweather-app_pipeline_PR-997/hercules/scripts/exregional_run_met_pcpcombine.sh: line 140: 10#: invalid integer constant (error token is "10#")
End exregional_run_met_pcpcombine.sh at Thu Jan  4 17:28:02 UTC 2024 with error code 1 (time elapsed: 00:00:03)

[mkavulich]

@MichaelLueken Thanks for letting me know, can you point me to the location of Jenkins tests on Hercules? I'm not sure where to find that information, is it in the Wiki somewhere?

[MichaelLueken]

@mkavulich - The Jenkins workspace for your test on Hercules is:

/work/noaa/epic/role-epic/jenkins/workspace/fs-srweather-app_pipeline_PR-997/hercules/expt_dirs/MET_verification_only_vx

At the bottom of the Contributor's Guide, there are paths to the Jenkins workspaces on the various machines.

[mkavulich]

@MichaelLueken For the record I am getting "Permission denied" errors when I attempt to view the contents of /work/noaa/epic/role-epic/jenkins/workspace/fs-srweather-app_pipeline_PR-997/hercules/expt_dirs/MET_verification_only_vx, can that be opened up in the future? I was able to replicate the error on my own so it's not hugely urgent

[MichaelLueken]

@MichaelLueken For the record I am getting "Permission denied" errors when I attempt to view the contents of /work/noaa/epic/role-epic/jenkins/workspace/fs-srweather-app_pipeline_PR-997/hercules/expt_dirs/MET_verification_only_vx, can that be opened up in the future? I was able to replicate the error on my own so it's not hugely urgent

@mkavulich - Thanks for bringing this to my attention. I have opened an issue with the EPIC Platform team to see if they can make the Jenkins workspace on Hercules readable for developers outside of the epic account.

[mkavulich]

@MichaelLueken This PR is ready for a re-test on Hercules.

For anyone curious about the details: This was a very strange confluence of circumstances revealing an old bug where some logic in exregional_run_met_pcpcombine.sh that was only used for forecast jobs (not for observations) needed to be moved inside an if-block to avoid utilizing undefined variables (specifically, the variable ENSMEM_INDX doesn't make sense for observation tasks since they are independent of the number of ensemble members). This didn't cause an error until I introduced my Octal bug fix...and even then, for some reason this is only failing on Hercules: tests on Derecho and Hera still pass. So something specific about the bash version (or other environment) on Hercules is revealing more problems than other machines.

[MichaelLueken]

@mkavulich - The WE2E tests successfully passed on Hercules:

---------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
custom_GFDLgrid__GFDLgrid_USE_NUM_CELLS_IN_FILENAMES_eq_FALSE_202  COMPLETE               8.54
grid_CONUS_25km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_202  COMPLETE              10.44
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_202  COMPLETE              30.58
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              18.92
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_2024011108  COMPLETE              26.41
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP_20240111083  COMPLETE              54.92
grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_  COMPLETE              14.94
grid_RRFS_NA_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP_20240111083520  COMPLETE              67.05
grid_SUBCONUS_Ind_3km_ics_NAM_lbcs_NAM_suite_GFS_v16_202401110835  COMPLETE              28.96
MET_verification_only_vx_20240111083521                            COMPLETE               0.22
specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS_20240111083523               COMPLETE               8.57
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             269.55

Merging this work now.