[develop] Integrate UW CLI tool for templater and remove external dependency.

christinaholtNOAA commented 9 months ago

DESCRIPTION OF CHANGES:

The workflow-tools package was initially integrated with SRW as an external repository under ush/python_utils. Since then, we have packaged the code as a conda package and it is now installed automatically on most platforms (WCOSS excluded, but with workarounds in place).

In this PR, I am removing the prior integration and leaning on the UW command line tools available from the conda package. For now, this involves calling the command line tools in a subprocess from Python code. We have an API under development that will replace this in the near future, so this will not likely be the final result for the Python-based scripts you see here.

Type of change

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

TESTS CONDUCTED:

[x] hera.intel
[ ] orion.intel
[ ] hercules.intel
[ ] cheyenne.intel
[ ] cheyenne.gnu
[ ] derecho.intel
[ ] gaea.intel
[ ] gaeac5.intel
[ ] jet.intel
[ ] wcoss2.intel
[ ] NOAA Cloud (indicate which platform)
[ ] Jenkins
[x] fundamental test suite
[ ] comprehensive tests (specify which if a subset was used)

DEPENDENCIES:

None

DOCUMENTATION:

None. UW Documentation is currently being updated to reflect changes in CLI tools.

CHECKLIST

[ ] My code follows the style guidelines in the Contributor's Guide
[ ] I have performed a self-review of my own code using the Code Reviewer's Guide
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] My changes need updates to the documentation. I have made corresponding changes to the documentation
[ ] My changes do not require updates to the documentation (explain).
[ ] My changes generate no new warnings
[ ] New and existing tests pass with my changes
[ ] Any dependent changes have been merged and published

MichaelLueken commented 8 months ago

The coverage tests were manually run on Derecho and all successfully passed:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
custom_ESGgrid_IndianOcean_6km                                     COMPLETE              23.35
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot     COMPLETE              37.49
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16                COMPLETE              44.89
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_HRRR           COMPLETE              29.37
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta    COMPLETE              17.90
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_HRRR                COMPLETE              40.68
nco_grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_timeoffset_suite_  COMPLETE              24.75
pregen_grid_orog_sfc_climo                                         COMPLETE              14.85
specify_template_filenames                                         COMPLETE              15.07
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             248.35

christinaholtNOAA commented 8 months ago

Just out of curiosity, what's the hang up on the Jenkins tests? I know Hera was down yday, but it seems bigger than that given they were kicked off Monday.

MichaelLueken commented 8 months ago

@christinaholtNOAA - The get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_netcdf_2022060112_48h test is failing on Jet in the make_ics and make_lbcs tasks (with terminate called after throwing an instance of 'std::bad_alloc' error messages).

While the tests were able to successfully complete after several repeated rewinds/boots, the run_post tasks all failed due to missing dynf000.nc and phyf000.nc files. I have relaunched the tests on Jet and Jet only has two core hours for the month, so we are regrettably running in windfall on the machine.

MichaelLueken commented 8 months ago

@christinaholtNOAA - Just wanting to give you a head's up, the current WE2E test runs on Jet look like they are successfully passing this time. No signs of DEAD in the WE2E_tests_20240110191405.yaml file and all of the tests have made it to the run_fcst task. I should be able to merge this PR later today or in the morning. The run directory, if you would like to check, is /lfs1/NAGAPE/epic/role.epic/jenkins/workspace/fs-srweather-app_pipeline_PR-994/jet/

MichaelLueken commented 8 months ago

The rerun of the WE2E tests on Jet have successfully passed:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
community                                                          COMPLETE              17.69
custom_ESGgrid                                                     COMPLETE              18.51
custom_ESGgrid_Great_Lakes_snow_8km                                COMPLETE              12.22
custom_GFDLgrid                                                    COMPLETE               9.86
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2021032018         COMPLETE               9.36
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_netcdf_2022060112_48h     COMPLETE              54.47
get_from_HPSS_ics_RAP_lbcs_RAP                                     COMPLETE              15.46
grid_RRFS_AK_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR                 COMPLETE             223.49
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot     COMPLETE              41.33
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2        COMPLETE               8.90
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta       COMPLETE             511.77
nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR       COMPLETE              10.72
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             933.78

ufs-community / ufs-srweather-app