[develop] Add integration test job

[EdwardSnyder-NOAA]

DESCRIPTION OF CHANGES:

This PR adds a test job to the workflow. It was originally written with pytest but because of some file naming issues, the python package unittest was used instead (see reasoning below). The test checks for the existence of netcdf files from the weather model. The hope is to expand the number of integration tests to eventually include regression testing.

The necessary scripts were added or modified to incorporate the integration job into the workflow. A wrapper script was also added.

unittest package reasoning: In order to pass arguments into a pytest script a conftest.py file needs to reside in the testing script directory or the directory above. Since this is a scheduler job, the conftest.py file will need to reside in ufs-srweather-app/scripts, but that would conflict with the NCO naming requirement, and storing conftest.py a directory above wouldn't make much sense either. Switching to unittest became the next best option.

Type of change

• [ ] Bug fix (non-breaking change which fixes an issue)
• [X] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
• [X] This change requires a documentation update

TESTS CONDUCTED:

• [ ] hera.intel
• [ ] orion.intel
• [ ] hercules.intel
• [ ] cheyenne.intel
• [ ] cheyenne.gnu
• [ ] derecho.intel
• [ ] gaea.intel
• [ ] gaeac5.intel
• [ ] jet.intel
• [ ] wcoss2.intel
• [X] NOAA Cloud (indicate which platform) ran fundamental on AWS: /contrib/Edward.Snyder/pytest/expt_dirs/fundamental
• [ ] Jenkins
• [X] fundamental test suite
• [ ] comprehensive tests (specify which if a subset was used)

DEPENDENCIES:

DOCUMENTATION:

Updated docs where it referred to the workflow tasks for the SRW App.

ISSUE:

CHECKLIST

• [X] My code follows the style guidelines in the Contributor's Guide
• [X] I have performed a self-review of my own code using the Code Reviewer's Guide
• [X] I have commented my code, particularly in hard-to-understand areas
• [x] My changes need updates to the documentation. I have made corresponding changes to the documentation
• [ ] My changes do not require updates to the documentation (explain).
• [X] My changes generate no new warnings
• [X] New and existing tests pass with my changes
• [X] Any dependent changes have been merged and published

LABELS (optional):

A Code Manager needs to add the following labels to this PR:

• [ ] Work In Progress
• [ ] bug
• [ ] enhancement
• [ ] documentation
• [ ] release
• [ ] high priority
• [ ] run_ci
• [ ] run_we2e_fundamental_tests
• [ ] run_we2e_comprehensive_tests
• [ ] Needs Cheyenne test
• [ ] Needs Jet test
• [ ] Needs Hera test
• [ ] Needs Orion test
• [ ] help wanted

CONTRIBUTORS (optional):

[MichaelLueken]

@EdwardSnyder-NOAA -

The WE2E test, deactivate_tasks, doesn't actually run the weather model. This causes the test to hang, since it is still trying to run the integration tests without the weather model. It would probably be best to go into tests/WE2E/test_configs/wflow_features/config.deactivate_tasks.yaml and add:

metatask_integration_test:

to cancel out the integration test for this specific WE2E.

[EdwardSnyder-NOAA]

@EdwardSnyder-NOAA -

The WE2E test, deactivate_tasks, doesn't actually run the weather model. This causes the test to hang, since it is still trying to run the integration tests without the weather model. It would probably be best to go into tests/WE2E/test_configs/wflow_features/config.deactivate_tasks.yaml and add:

metatask_integration_test:

to cancel out the integration test for this specific WE2E.

Thanks for catching that! I made the change as requested.

Eddie

[MichaelLueken]

All tests have successfully passed, except for two tests on Hera Intel - grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2 and pregen_grid_orog_sfc_climo. Both tests failed in run_fcst. Rocotorewind/rocotoboot have been used on these two tests. Once complete, I will add the updated summary and merge this PR.

[MichaelLueken]

Utilizing the Rocky8 nodes on Hera, the last two tests that needed to pass to move forward with this PR completed successfully:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
custom_ESGgrid_Peru_12km_20240305060102                            COMPLETE              18.10
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2019061200_2024030  COMPLETE               7.64
get_from_HPSS_ics_GDAS_lbcs_GDAS_fmt_netcdf_2022040400_ensemble_2  COMPLETE             755.64
get_from_HPSS_ics_HRRR_lbcs_RAP_20240305060107                     COMPLETE              15.27
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20240  COMPLETE               6.55
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20  COMPLETE              14.89
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_RAP_20240305060111  COMPLETE              10.80
grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2_20240  COMPLETE               8.15
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_202403  COMPLETE             232.28
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_20240305  COMPLETE             301.03
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_HRRR_202403050  COMPLETE             324.97
pregen_grid_orog_sfc_climo_20240305060118                          COMPLETE               7.09
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE            1702.41

Merging this PR now.