ufs-community / ufs-srweather-app

UFS Short-Range Weather Application
Other
55 stars 116 forks source link

The nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR WE2E test fails when ran using the comprehensive suite #974

Closed MichaelLueken closed 9 months ago

MichaelLueken commented 10 months ago

Expected behavior

All comprehensive tests should successfully pass on a given machine.

Current behavior

The nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR WE2E test fails in the make_ics task on all machines.

Machines affected

This failure is occurring on all Tier-1 RDHPCS machines (Derecho, Gaea, Gaea C5, Hera, Hercules, Jet, NOAA Cloud, and Orion).

Steps To Reproduce

  1. Clone the HEAD of SRW App's develop branch.
  2. Build the App
  3. Run the comprehensive WE2E test suite
  4. See the failure in nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR.

Additional Information (optional)

If the nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR test is ran by itself, or on Jet as part of the coverage tests, this test will successfully pass. Additionally, once the failure in make_ics occurs, the use of rocotorewind and rocotoboot on the failed task will continue to result in failure.

Output (optional)

PET00 ESMF_LogFile OUTPUT_file_contents