[Release/public-v2.2.0] Update data paths to v2p2

ufs-community / ufs-srweather-app

UFS Short-Range Weather Application

Other

56 stars 119 forks source link

[Release/public-v2.2.0] Update data paths to v2p2 #937

Closed EdwardSnyder-NOAA closed 1 year ago

EdwardSnyder-NOAA commented 1 year ago

DESCRIPTION OF CHANGES:

Update data paths to point the release staged data location for all supported platforms.

Type of change

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

TESTS CONDUCTED:

[ ] hera.intel
[ ] orion.intel
[ ] hercules.intel
[ ] cheyenne.intel
[ ] cheyenne.gnu
[ ] derecho.intel
[ ] gaea.intel
[ ] gaeac5.intel
[ ] jet.intel
[ ] wcoss2.intel
[x] NOAA Cloud (indicate which platform: aws)
[ ] Jenkins
[ ] fundamental test suite
[ ] comprehensive tests (specify which if a subset was used)

DEPENDENCIES:

DOCUMENTATION:

ISSUE:

CHECKLIST

[x] My code follows the style guidelines in the Contributor's Guide
[x] I have performed a self-review of my own code using the Code Reviewer's Guide
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] My changes need updates to the documentation. I have made corresponding changes to the documentation
[x] My changes do not require updates to the documentation (explain).
[ ] My changes generate no new warnings
[ ] New and existing tests pass with my changes
[x] Any dependent changes have been merged and published

LABELS (optional):

A Code Manager needs to add the following labels to this PR:

[ ] Work In Progress
[ ] bug
[ ] enhancement
[ ] documentation
[x] release
[ ] high priority
[ ] run_ci
[ ] run_we2e_fundamental_tests
[ ] run_we2e_comprehensive_tests
[ ] Needs Cheyenne test
[ ] Needs Jet test
[ ] Needs Hera test
[ ] Needs Orion test
[ ] help wanted

CONTRIBUTORS (optional):

MichaelLueken commented 1 year ago

Submitting comprehensive Jenkins tests now. Please note that the comprehensive tests can only be launched manually in the Jenkins API. The run_we2e_coverage_tests label is still used to kick off the comprehensive tests.

MichaelLueken commented 1 year ago

The pipeline for the comprehensive tests for this PR can be found - https://jenkins.epic.oarcloud.noaa.gov/blue/organizations/jenkins/ufs-srweather-app%2Fpipeline/detail/PR-937/2/pipeline

mkavulich commented 1 year ago

@natalie-perlin I think I understand where your confusion is coming from. There are various old conventions for testing that are not explained anywhere, and because of this have not been followed by everyone. There also appear to be some outdated and unnecessary variables.

The data: stanzas are meant to be pointing to various useful sets of staged data that may exist on platforms for use in experiments (for example, on Jet there are various data archives) for a wide range of potential dates. They should not be pointing to WE2E test data, which is meant to only be used for WE2E tests with very specific staged data for very specific dates. I did not notice when these were added (looks like it was #674, #913, and the various "new platform" PRs), otherwise I would have raised these points then.

To clarify the need for TEST_* variables and why data: should not point to the staged test data, the WE2E tests needed a way to test whether the variables that are meant for users to point to data: EXTRN_MDL_SOURCE_BASEDIR for initial and lateral boundary conditions, *_OBS_DIR for various types of obs data for verification experiments, VX_FCST_INPUT_BASEDIR for staging stand-alone forecast files for verification experiments, and PREGEN_BASEDIR for pre-generated static forecast files from make_grid, make_orog, and make_sfc_climo) All those "TEST_*" variables are used by the WE2E script to set those user-specified variables, so that their behavior can be tested and verified as working properly. Having the data: stanza point to these WE2E-specific data locations means that these checks can be bypassed, and so the tests are not functioning as intended!

I believe TEST_ALT_EXTRN_MDL_SYSBASEDIR_ICS and TEST_ALT_EXTRN_MDL_SYSBASEDIR_LBCS are obsolete and can be removed. I'm not sure what these were originally supposed to be testing; @gsketefian may remember since he designed much of the initial testing system.

We should probably open an issue to discuss and resolve these problems.

natalie-perlin commented 1 year ago

@EdwardSnyder-NOAA - please see issue https://github.com/ufs-community/ufs-srweather-app/issues/925

There is a difference between the use of the variable TEST_EXTRN_MDL_SOURCE_BASEDIR and use of data section in the end of the machine file, i.e.:

data:
  ics_lbcs:

When test or suite/tests from WE2E are run from ./tests/WE2E/ directory, their configuration includes using the variable TEST_EXTRN_MDL_SOURCE_BASEDIR to determine data locations for the get_ics and get_lbcs tasks, which is then stored as EXTRN_MDL_SOURCE_BASEDIR variable in config.yaml file

For running a community test case on a Tier 1 system from ./ush directory using config.yaml script (copying from config.community.yaml), and launching generate_FV3LAM_wflow.py (copying from config.community.yaml), a variable TEST_EXTRN_MDL_SOURCE_BASEDIR is not used to set prestaged data locations for get_ics and get_lbcs tasks.

The prestaged locations need to be in a "data: ics_lbcs" section of the .yaml file for the generated community case to search for data locally ("disk"), and not to attempt to pull it from hpss or aws.

MichaelLueken commented 1 year ago

@EdwardSnyder-NOAA - The Hercules comprehensive tests failed (I suspect the Orion tests will fail as well). There are no orography fix files in: /work/noaa/epic/role-epic/contrib/UFS_SRW_data/v2p2/FV3LAM_pregen/RRFS_CONUS_3km which is causing the tests to fail while attempting to link the C*_oro_data_ss.tile7.halo0.nc file.

I'll let you know if or when other tests fail.

MichaelLueken commented 1 year ago

@EdwardSnyder-NOAA - As noted above, the Orion comprehensive tests also failed since the necessary orography fix files are not available.

MichaelLueken commented 1 year ago

@EdwardSnyder-NOAA - The comprehensive tests also failed on Hera (both Intel and GNU), because C*_oro_data_ss.tile7.halo0.nc wasn't available in /scratch1/NCEPDEV/nems/role.epic/UFS_SRW_data/v2p2/FV3LAM_pregen/RRFS_CONUS_3km/

MichaelLueken commented 1 year ago

@EdwardSnyder-NOAA - There were failures in both Hera Intel and Hera GNU comprehensive tests. The following tests errored out due to issues with rocoto:

grid_CONUS_3km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta,
grid_RRFS_AK_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot,
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP, and
long_fcst

The following tests failed:

grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_RAP- Both run_MET_GridStat_vx_REFC_mem000 and run_MET_GridStat_vx_RETOP_mem000 tasks failed with the following error message Could not find OBS_INPUT files under /scratch1/NCEPDEV/nems/role.epic/UFS_SRW_data/v2p2/obs_data/mrms/proc. Both tasks successfully found the files at the start of the log:
```
Found file (fp) for the current forecast hour (fhr; relative to the cycle
date cdate):
fhr = "6"
cdate = "2019061518"
fp = "/scratch1/NCEPDEV/nems/role.epic/UFS_SRW_data/v2p2/obs_data/mrms/proc/20190616/EchoTop_18_00.50_20190616-000000.grib2"
```
grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_WoFS_v0 - Both run_MET_GridStat_vx_REFC_mem000 and run_MET_GridStat_vx_RETOP_mem000 failed, similar to above, due to not finding the observation files, even though they are present in v2p2.
MET_ensemble_verification_only_vx - run_MET_GridStat_vx_REFC_mem001, run_MET_GridStat_vx_RETOP_mem001, run_MET_GridStat_vx_REFC_mem002, run_MET_GridStat_vx_RETOP_mem002, run_MET_EnsembleStat_vx_REFC, run_MET_EnsembleStat_vx_RETOP, run_MET_GridStat_vx_ensprob_REFC, and run_MET_GridStat_vx_ensprob_RETOP tasks failed, similar to above, due to not finding the observation files, even though they are present in v2p2.
MET_verification_only_vx - run_MET_GridStat_vx_REFC_mem000 and run_MET_GridStat_vx_RETOP_mem000 tasks failed, similar to above, due to not finding the observation file, even though they are present in v2p2.
nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR - Only make_ics_mem000 failed. Most likely, a rerun of the failed task would work.

How would you like to proceed with the testing now? I should note that the verification issues were also being encountered during manual runs of the comprehensive testing suite on Derecho.

MichaelLueken commented 1 year ago

The following is a more detailed error message for the failed verification tasks:

10/12 23:25:15.030 metplus (command_builder.py:265) ERROR: (command_builder.py:808) Could not find OBS_INPUT files under /scratch1/NCEPDEV/nems/role.epic/UFS_SRW_data/v2p2/obs_data/mrms/proc within range [-300,300] using template {valid?fmt=%Y%m%d}/MergedReflectivityQCComposite_00.50_{valid?fmt=%Y%m%d}-{valid?fmt=%H%M%S}.grib2
10/12 23:25:15Z run_MET_GridStat_vx_REFC_mem000-metplus: ERROR:  (command_builder.py:808) Could not find OBS_INPUT files under /scratch1/NCEPDEV/nems/role.epic/UFS_SRW_data/v2p2/obs_data/mrms/proc within range [-300,300] using template {valid?fmt=%Y%m%d}/MergedReflectivityQCComposite_00.50_{valid?fmt=%Y%m%d}-{valid?fmt=%H%M%S}.grib2
10/12 23:25:15.030 metplus (command_builder.py:265) ERROR: (command_builder.py:548) Could not find observation file
10/12 23:25:15Z run_MET_GridStat_vx_REFC_mem000-metplus: ERROR:  (command_builder.py:548) Could not find observation file

MichaelLueken commented 1 year ago

@EdwardSnyder-NOAA - I see that the verification observational data is a link from v2p0, rather than a complete copy, for v2p2. It is possible that the script doesn't view that the data is available in that location, because the data is actually located in /scratch1/NCEPDEV/nems/role.epic/UFS_SRW_data/v2p0/obs_data/mrms/proc, rather than /scratch1/NCEPDEV/nems/role.epic/UFS_SRW_data/v2p2/obs_data/mrms/proc.

gspetro-NOAA commented 1 year ago

@natalie-perlin It seems like there is no need to set the data: section of the machine files at all to run a general experiment, and based on @mkavulich 's comment, the data: stanza should not be used in testing. The instructions in the documentation tell users how to set the ICS/LBCS data locations for their experiment using USE_USER_STAGED_EXTRN_FILES: true and setting the paths to EXTRN_MDL_SOURCE_BASEDIR_[ICS/LBCS]. That data could be prestaged system data or data for non-standard dates that the user has acquired some other way. The data: section is more of a convenience for users with multiple data sets to track (they can specify data locations there instead), but it doesn't need to be used at all for a general user experiment. Issue #925 seems to deal with the fact that an experiment won't check two disk locations. It checks one disk location, and if the data isn't there, it moves on to other sources. That seems to be the core issue that needs resolving, and it sounds like one solution is not to point the data: section to staged test data (instead, we should be using the TEST_* variables). Having @EdwardSnyder-NOAA add a bunch of paths in the data: section of machine files is likely to mess with the effectiveness/integrity of our WE2E testing, which would be a serious problem.

MichaelLueken commented 1 year ago

Manually rerunning all tests that either failed or were never launched have passed, with the exception of nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
2020_CAD                                                           COMPLETE              55.62
community                                                          COMPLETE              38.70
custom_ESGgrid                                                     COMPLETE              20.20
custom_ESGgrid_Central_Asia_3km                                    COMPLETE              37.08
custom_ESGgrid_Great_Lakes_snow_8km                                COMPLETE              17.31
custom_ESGgrid_IndianOcean_6km                                     COMPLETE              27.47
custom_ESGgrid_NewZealand_3km                                      COMPLETE              86.31
custom_ESGgrid_Peru_12km                                           COMPLETE              26.26
custom_ESGgrid_SF_1p1km                                            COMPLETE             265.76
custom_GFDLgrid__GFDLgrid_USE_NUM_CELLS_IN_FILENAMES_eq_FALSE      COMPLETE              10.64
custom_GFDLgrid                                                    COMPLETE              10.24
deactivate_tasks                                                   COMPLETE               1.07
get_from_AWS_ics_GEFS_lbcs_GEFS_fmt_grib2_2022040400_ensemble_2me  COMPLETE            1265.92
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2019061200          COMPLETE               9.57
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2019061200         COMPLETE              11.57
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2021032018         COMPLETE              11.90
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_netcdf_2022060112_48h     COMPLETE             111.17
get_from_HPSS_ics_GDAS_lbcs_GDAS_fmt_netcdf_2022040400_ensemble_2  COMPLETE            1290.88
get_from_HPSS_ics_GSMGFS_lbcs_GSMGFS                               COMPLETE               9.56
get_from_HPSS_ics_HRRR_lbcs_RAP                                    COMPLETE              22.25
get_from_HPSS_ics_RAP_lbcs_RAP                                     COMPLETE              26.59
get_from_NOMADS_ics_FV3GFS_lbcs_FV3GFS                             COMPLETE              18.73
grid_CONUS_25km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16      COMPLETE              12.56
grid_CONUS_3km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta   COMPLETE             411.33
grid_RRFS_AK_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot        COMPLETE             258.29
grid_RRFS_AK_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR                 COMPLETE             289.72
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP              COMPLETE              83.34
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot     COMPLETE              52.17
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR             COMPLETE              45.75
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta      COMPLETE              43.56
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2        COMPLETE               8.74
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot     COMPLETE              20.74
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              19.76
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR             COMPLETE              26.76
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP              COMPLETE              52.52
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta      COMPLETE              27.94
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_RAP                 COMPLETE              14.08
grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2        COMPLETE               9.49
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16                COMPLETE              39.69
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_RRFS_v1beta            COMPLETE              24.45
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2         COMPLETE             404.00
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson  COMPLETE             534.54
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16           COMPLETE             681.72
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR              COMPLETE             623.10
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta       COMPLETE             590.40
grid_RRFS_CONUScompact_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16   COMPLETE              46.38
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_HRRR           COMPLETE              41.57
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta    COMPLETE              38.30
grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16   COMPLETE              14.91
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR          COMPLETE              39.09
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta    COMPLETE              12.78
grid_RRFS_CONUScompact_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16    COMPLETE             550.23
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_HRRR            COMPLETE             551.52
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta     COMPLETE             519.39
grid_RRFS_NA_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP                 COMPLETE             108.79
grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_WoFS_v0         COMPLETE              21.75
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_HRRR                COMPLETE              25.08
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0              COMPLETE              20.98
grid_SUBCONUS_Ind_3km_ics_NAM_lbcs_NAM_suite_GFS_v16               COMPLETE              50.48
grid_SUBCONUS_Ind_3km_ics_RAP_lbcs_RAP_suite_RRFS_v1beta_plot      COMPLETE              11.88
long_fcst                                                          COMPLETE              73.74
MET_ensemble_verification_only_vx                                  COMPLETE               1.29
MET_ensemble_verification_only_vx_time_lag                         COMPLETE               7.97
MET_verification_only_vx                                           COMPLETE               0.25
nco                                                                COMPLETE              17.33
nco_ensemble                                                       COMPLETE              82.02
nco_grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16      COMPLETE              63.55
nco_grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_timeoffset_suite_  COMPLETE              21.86
nco_grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thom  COMPLETE             523.37
nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR       DEAD                   1.34
pregen_grid_orog_sfc_climo                                         COMPLETE               9.93
specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS                              COMPLETE               9.57
specify_template_filenames                                         COMPLETE              10.20
----------------------------------------------------------------------------------------------------
Total                                                              DEAD               10525.00

Comprehensive testing on Hera GNU is now complete.

MichaelLueken commented 1 year ago

Running the failed test, nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR, it successfully passes for Hera GNU:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR       COMPLETE              19.67
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE              19.67

EdwardSnyder-NOAA commented 1 year ago

Some of the Hera GNU tests were failing because the C3359_oro_data_ls.tile7.halo0.nc and C3359_oro_data_ss.tile7.halo0.nc were missing under /scratch1/NCEPDEV/nems/role.epic/UFS_SRW_data/v2p2/FV3LAM_pregen/RRFS_CONUS_3km or the /scratch1/NCEPDEV/nems/role.epic/UFS_SRW_data/v2p2/obs_data/mrms/proc/2019061* directories were linked. I copied over the two missing files, and unlinked the obs_data directories and linked the files instead. Doing this resulted in the Hera GNU tests to pass. These changes were made to the other T1 platforms.

MichaelLueken commented 1 year ago

The comprehensive tests have completed for Hera Intel:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
2020_CAD                                                           COMPLETE              34.61
community                                                          COMPLETE              18.64
custom_ESGgrid                                                     COMPLETE              15.14
custom_ESGgrid_Central_Asia_3km                                    COMPLETE              25.70
custom_ESGgrid_Great_Lakes_snow_8km                                COMPLETE              12.16
custom_ESGgrid_IndianOcean_6km                                     COMPLETE              19.03
custom_ESGgrid_NewZealand_3km                                      COMPLETE              53.08
custom_ESGgrid_Peru_12km                                           COMPLETE              18.56
custom_ESGgrid_SF_1p1km                                            COMPLETE             158.95
custom_GFDLgrid__GFDLgrid_USE_NUM_CELLS_IN_FILENAMES_eq_FALSE      COMPLETE               8.09
custom_GFDLgrid                                                    COMPLETE               7.66
deactivate_tasks                                                   COMPLETE               1.01
get_from_AWS_ics_GEFS_lbcs_GEFS_fmt_grib2_2022040400_ensemble_2me  COMPLETE             755.18
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2019061200          COMPLETE               6.38
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2019061200         COMPLETE               9.26
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2021032018         COMPLETE               8.96
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_netcdf_2022060112_48h     COMPLETE              58.65
get_from_HPSS_ics_GDAS_lbcs_GDAS_fmt_netcdf_2022040400_ensemble_2  COMPLETE             768.90
get_from_HPSS_ics_GSMGFS_lbcs_GSMGFS                               COMPLETE               6.83
get_from_HPSS_ics_HRRR_lbcs_RAP                                    COMPLETE              14.31
get_from_HPSS_ics_RAP_lbcs_RAP                                     COMPLETE              17.28
get_from_NOMADS_ics_FV3GFS_lbcs_FV3GFS                             COMPLETE              12.87
grid_CONUS_25km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16      COMPLETE              10.14
grid_CONUS_3km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta   COMPLETE             263.87
grid_RRFS_AK_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot        COMPLETE             138.17
grid_RRFS_AK_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR                 COMPLETE             194.40
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP              COMPLETE              25.76
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot     COMPLETE              31.22
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR             COMPLETE              29.16
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta      COMPLETE              29.27
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2        COMPLETE               6.97
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot     COMPLETE              14.79
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              14.04
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR             COMPLETE              16.48
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP              COMPLETE              48.08
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta      COMPLETE              21.27
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_RAP                 COMPLETE              10.46
grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2        COMPLETE               7.43
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16                COMPLETE              20.81
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_RRFS_v1beta            COMPLETE              15.92
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2         COMPLETE             238.53
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson  COMPLETE             332.98
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16           COMPLETE             305.20
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR              COMPLETE             376.46
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta       COMPLETE             368.78
grid_RRFS_CONUScompact_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16   COMPLETE              28.51
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_HRRR           COMPLETE              26.75
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta    COMPLETE              25.63
grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16   COMPLETE              11.99
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR          COMPLETE              27.47
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta    COMPLETE               9.04
grid_RRFS_CONUScompact_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16    COMPLETE             275.26
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_HRRR            COMPLETE             326.60
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta     COMPLETE             330.02
grid_RRFS_NA_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP                 COMPLETE              69.54
grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_WoFS_v0         COMPLETE              14.62
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_HRRR                COMPLETE              16.94
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0              COMPLETE              13.99
grid_SUBCONUS_Ind_3km_ics_NAM_lbcs_NAM_suite_GFS_v16               COMPLETE              20.08
grid_SUBCONUS_Ind_3km_ics_RAP_lbcs_RAP_suite_RRFS_v1beta_plot      COMPLETE               8.73
long_fcst                                                          COMPLETE              54.60
MET_ensemble_verification_only_vx                                  COMPLETE               1.01
MET_ensemble_verification_only_vx_time_lag                         COMPLETE               3.19
MET_verification_only_vx                                           COMPLETE               0.21
nco                                                                COMPLETE               8.16
nco_ensemble                                                       COMPLETE              64.17
nco_grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16      COMPLETE              27.03
nco_grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_timeoffset_suite_  COMPLETE              12.57
nco_grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thom  COMPLETE             323.30
nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR       DEAD                   1.32
pregen_grid_orog_sfc_climo                                         COMPLETE               7.22
specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS                              COMPLETE               6.38
specify_template_filenames                                         COMPLETE               8.14
----------------------------------------------------------------------------------------------------
Total                                                              DEAD                6273.91

Once again, nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR is the only failure.

MichaelLueken commented 1 year ago

The comprehensive tests have completed on Jet:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
2020_CAD                                                           COMPLETE              51.79
community                                                          COMPLETE              18.61
custom_ESGgrid                                                     COMPLETE              19.75
custom_ESGgrid_Central_Asia_3km                                    COMPLETE              36.64
custom_ESGgrid_Great_Lakes_snow_8km                                COMPLETE              14.38
custom_ESGgrid_IndianOcean_6km                                     COMPLETE              17.34
custom_ESGgrid_NewZealand_3km                                      COMPLETE              73.46
custom_ESGgrid_Peru_12km                                           COMPLETE              23.89
custom_ESGgrid_SF_1p1km                                            COMPLETE             253.67
custom_GFDLgrid__GFDLgrid_USE_NUM_CELLS_IN_FILENAMES_eq_FALSE      COMPLETE              10.32
custom_GFDLgrid                                                    COMPLETE               9.43
deactivate_tasks                                                   COMPLETE               0.93
get_from_AWS_ics_GEFS_lbcs_GEFS_fmt_grib2_2022040400_ensemble_2me  COMPLETE            1029.88
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2019061200          COMPLETE               8.53
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2019061200         COMPLETE              15.93
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2021032018         COMPLETE              18.26
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_netcdf_2022060112_48h     COMPLETE              68.91
get_from_HPSS_ics_GDAS_lbcs_GDAS_fmt_netcdf_2022040400_ensemble_2  COMPLETE            1048.35
get_from_HPSS_ics_GSMGFS_lbcs_GSMGFS                               COMPLETE              11.06
get_from_HPSS_ics_HRRR_lbcs_RAP                                    COMPLETE              16.28
get_from_HPSS_ics_RAP_lbcs_RAP                                     COMPLETE              20.34
get_from_NOMADS_ics_FV3GFS_lbcs_FV3GFS                             COMPLETE              13.27
grid_CONUS_25km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16      COMPLETE              14.76
grid_CONUS_3km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta   COMPLETE             377.87
grid_RRFS_AK_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot        COMPLETE             168.46
grid_RRFS_AK_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR                 COMPLETE             232.95
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP              COMPLETE              34.55
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot     COMPLETE              42.04
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR             COMPLETE              39.51
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta      COMPLETE              41.46
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2        COMPLETE               7.65
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot     COMPLETE              15.77
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              14.84
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR             COMPLETE              17.33
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP              COMPLETE              45.57
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta      COMPLETE              20.08
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_RAP                 COMPLETE              12.04
grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2        COMPLETE               6.87
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16                COMPLETE              21.84
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_RRFS_v1beta            COMPLETE              15.98
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2         COMPLETE             365.57
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson  COMPLETE             463.16
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16           COMPLETE             442.15
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR              COMPLETE             551.64
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta       COMPLETE             561.13
grid_RRFS_CONUScompact_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16   COMPLETE              35.41
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_HRRR           COMPLETE              34.65
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta    COMPLETE              33.74
grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16   COMPLETE              11.26
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR          COMPLETE              28.83
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta    COMPLETE               9.42
grid_RRFS_CONUScompact_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16    COMPLETE             375.91
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_HRRR            COMPLETE             446.56
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta     COMPLETE             441.20
grid_RRFS_NA_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP                 COMPLETE              98.52
grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_WoFS_v0         COMPLETE              22.43
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_HRRR                COMPLETE              27.51
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0              COMPLETE              22.18
grid_SUBCONUS_Ind_3km_ics_NAM_lbcs_NAM_suite_GFS_v16               COMPLETE              32.69
grid_SUBCONUS_Ind_3km_ics_RAP_lbcs_RAP_suite_RRFS_v1beta_plot      COMPLETE              14.23
long_fcst                                                          COMPLETE              96.15
MET_ensemble_verification_only_vx                                  COMPLETE               1.44
MET_ensemble_verification_only_vx_time_lag                         COMPLETE               5.22
MET_verification_only_vx                                           COMPLETE               0.30
nco                                                                COMPLETE               7.94
nco_ensemble                                                       COMPLETE              76.29
nco_grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16      COMPLETE              37.40
nco_grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_timeoffset_suite_  COMPLETE              12.83
nco_grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thom  COMPLETE             439.15
nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR       DEAD                   1.52
pregen_grid_orog_sfc_climo                                         COMPLETE              10.35
specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS                              COMPLETE               7.38
specify_template_filenames                                         COMPLETE               9.88
----------------------------------------------------------------------------------------------------
Total                                                              DEAD                8634.63

MichaelLueken commented 1 year ago

The Gaea C5 tests have completed:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
2020_CAD                                                           COMPLETE              35.89
community                                                          COMPLETE              43.40
custom_ESGgrid                                                     COMPLETE              14.55
custom_ESGgrid_Central_Asia_3km                                    COMPLETE              34.10
custom_ESGgrid_Great_Lakes_snow_8km                                DEAD                   2.46
custom_ESGgrid_IndianOcean_6km                                     COMPLETE              23.25
custom_ESGgrid_NewZealand_3km                                      COMPLETE              47.59
custom_ESGgrid_Peru_12km                                           COMPLETE              22.24
custom_ESGgrid_SF_1p1km                                            COMPLETE             159.12
custom_GFDLgrid__GFDLgrid_USE_NUM_CELLS_IN_FILENAMES_eq_FALSE      COMPLETE              10.35
custom_GFDLgrid                                                    COMPLETE               9.14
deactivate_tasks                                                   COMPLETE               0.92
get_from_AWS_ics_GEFS_lbcs_GEFS_fmt_grib2_2022040400_ensemble_2me  COMPLETE             684.75
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2019061200          DEAD                   0.96
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2019061200         DEAD                   0.94
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2021032018         DEAD                   1.00
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_netcdf_2022060112_48h     DEAD                   1.06
get_from_HPSS_ics_GDAS_lbcs_GDAS_fmt_netcdf_2022040400_ensemble_2  DEAD                   3.41
get_from_HPSS_ics_GSMGFS_lbcs_GSMGFS                               DEAD                   0.94
get_from_HPSS_ics_HRRR_lbcs_RAP                                    DEAD                   0.99
get_from_HPSS_ics_RAP_lbcs_RAP                                     DEAD                   1.01
get_from_NOMADS_ics_FV3GFS_lbcs_FV3GFS                             COMPLETE              23.38
grid_CONUS_25km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16      COMPLETE              15.68
grid_CONUS_3km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta   COMPLETE             243.97
grid_RRFS_AK_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot        COMPLETE             129.90
grid_RRFS_AK_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR                 COMPLETE             171.18
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP              COMPLETE              30.48
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot     COMPLETE              34.36
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR             COMPLETE              30.29
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta      COMPLETE              30.37
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2        COMPLETE              13.62
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot     COMPLETE              27.95
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              26.53
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR             COMPLETE              39.51
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP              COMPLETE              73.34
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta      COMPLETE              41.72
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_RAP                 COMPLETE              18.99
grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2        COMPLETE              12.43
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16                COMPLETE              46.33
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_RRFS_v1beta            COMPLETE              36.72
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2         COMPLETE             242.21
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson  COMPLETE             314.58
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16           COMPLETE             328.08
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR              COMPLETE             344.43
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta       COMPLETE             344.22
grid_RRFS_CONUScompact_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16   COMPLETE              30.18
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_HRRR           COMPLETE              28.36
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta    COMPLETE              27.98
grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16   COMPLETE              18.94
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR          COMPLETE              31.81
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta    COMPLETE              18.04
grid_RRFS_CONUScompact_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16    COMPLETE             262.79
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_HRRR            COMPLETE             275.07
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta     COMPLETE             278.02
grid_RRFS_NA_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP                 COMPLETE              77.17
grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_WoFS_v0         COMPLETE              33.19
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_HRRR                COMPLETE              40.98
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0              COMPLETE              32.40
grid_SUBCONUS_Ind_3km_ics_NAM_lbcs_NAM_suite_GFS_v16               COMPLETE              52.09
grid_SUBCONUS_Ind_3km_ics_RAP_lbcs_RAP_suite_RRFS_v1beta_plot      COMPLETE              17.09
long_fcst                                                          COMPLETE              64.02
MET_ensemble_verification_only_vx                                  COMPLETE               1.06
MET_ensemble_verification_only_vx_time_lag                         DEAD                   0.12
MET_verification_only_vx                                           COMPLETE               0.21
nco                                                                COMPLETE              21.60
nco_ensemble                                                       COMPLETE             106.54
nco_grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16      COMPLETE              32.59
nco_grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_timeoffset_suite_  COMPLETE              25.82
nco_grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thom  COMPLETE             311.10
nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR       DEAD                   1.16
pregen_grid_orog_sfc_climo                                         COMPLETE              12.46
specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS                              COMPLETE              12.68
specify_template_filenames                                         COMPLETE              12.86
----------------------------------------------------------------------------------------------------
Total                                                              DEAD                5544.67

The only failures are related to pulling data from HPSS (which isn't available on Gaea C5) and nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR, which fails on all machines, but will run if run separately.

MichaelLueken commented 1 year ago

The comprehensive tests have completed on Hercules:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
2020_CAD                                                           COMPLETE              32.23
community                                                          COMPLETE              32.49
custom_ESGgrid                                                     COMPLETE              14.27
custom_ESGgrid_Central_Asia_3km                                    COMPLETE              23.15
custom_ESGgrid_IndianOcean_6km                                     COMPLETE              17.97
custom_ESGgrid_NewZealand_3km                                      COMPLETE              46.86
custom_ESGgrid_Peru_12km                                           COMPLETE              17.34
custom_ESGgrid_SF_1p1km                                            COMPLETE             139.08
custom_GFDLgrid__GFDLgrid_USE_NUM_CELLS_IN_FILENAMES_eq_FALSE      COMPLETE               7.53
custom_GFDLgrid                                                    COMPLETE               7.32
deactivate_tasks                                                   COMPLETE               0.80
get_from_AWS_ics_GEFS_lbcs_GEFS_fmt_grib2_2022040400_ensemble_2me  COMPLETE             635.43
get_from_NOMADS_ics_FV3GFS_lbcs_FV3GFS                             COMPLETE              15.98
grid_CONUS_25km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16      COMPLETE               9.99
grid_CONUS_3km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta   COMPLETE             208.44
grid_RRFS_AK_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot        COMPLETE             114.42
grid_RRFS_AK_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR                 COMPLETE             147.08
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP              COMPLETE              26.44
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot     COMPLETE              31.97
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR             COMPLETE              30.85
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta      COMPLETE              31.04
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2        COMPLETE               8.71
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot     COMPLETE              18.98
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              18.04
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR             COMPLETE              25.93
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP              COMPLETE              60.95
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta      COMPLETE              30.59
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_RAP                 COMPLETE              15.12
grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2        COMPLETE               8.76
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16                COMPLETE              32.12
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_RRFS_v1beta            COMPLETE              27.46
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2         COMPLETE             205.37
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson  COMPLETE             293.87
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16           COMPLETE             283.93
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR              COMPLETE             347.52
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta       COMPLETE             313.96
grid_RRFS_CONUScompact_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16   COMPLETE              29.37
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_HRRR           COMPLETE              29.60
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta    COMPLETE              25.75
grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16   COMPLETE              11.94
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR          COMPLETE              44.61
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta    COMPLETE              12.30
grid_RRFS_CONUScompact_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16    COMPLETE             268.38
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_HRRR            COMPLETE             290.68
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta     COMPLETE             290.51
grid_RRFS_NA_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP                 COMPLETE              71.27
grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_WoFS_v0         COMPLETE              23.74
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_HRRR                COMPLETE              25.17
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0              COMPLETE              22.67
grid_SUBCONUS_Ind_3km_ics_NAM_lbcs_NAM_suite_GFS_v16               COMPLETE              28.82
grid_SUBCONUS_Ind_3km_ics_RAP_lbcs_RAP_suite_RRFS_v1beta_plot      COMPLETE              11.55
MET_ensemble_verification_only_vx                                  COMPLETE               1.24
MET_verification_only_vx                                           COMPLETE               0.23
nco                                                                COMPLETE              12.74
nco_ensemble                                                       COMPLETE              84.63
nco_grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16      COMPLETE              28.15
nco_grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_timeoffset_suite_  COMPLETE              17.19
nco_grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thom  COMPLETE             288.61
nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR       DEAD                   1.02
pregen_grid_orog_sfc_climo                                         COMPLETE               9.51
specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS                              COMPLETE               8.46
specify_template_filenames                                         COMPLETE               9.80
----------------------------------------------------------------------------------------------------
Total                                                              DEAD                4929.93

MichaelLueken commented 1 year ago

The comprehensive tests completed on Orion:

--------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
2020_CAD                                                           COMPLETE              31.85
community                                                          COMPLETE              17.60
custom_ESGgrid                                                     COMPLETE              13.34
custom_ESGgrid_Central_Asia_3km                                    COMPLETE              24.71
custom_ESGgrid_IndianOcean_6km                                     COMPLETE              17.53
custom_ESGgrid_NewZealand_3km                                      COMPLETE              57.24
custom_ESGgrid_Peru_12km                                           COMPLETE              20.00
custom_ESGgrid_SF_1p1km                                            COMPLETE             160.21
custom_GFDLgrid__GFDLgrid_USE_NUM_CELLS_IN_FILENAMES_eq_FALSE      COMPLETE               9.29
custom_GFDLgrid                                                    COMPLETE               8.85
deactivate_tasks                                                   COMPLETE               1.37
get_from_AWS_ics_GEFS_lbcs_GEFS_fmt_grib2_2022040400_ensemble_2me  COMPLETE             761.55
get_from_NOMADS_ics_FV3GFS_lbcs_FV3GFS                             DEAD                   1.09
grid_CONUS_25km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16      COMPLETE              10.37
grid_CONUS_3km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta   COMPLETE             257.70
grid_RRFS_AK_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot        COMPLETE             137.66
grid_RRFS_AK_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR                 COMPLETE             181.82
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP              COMPLETE              26.72
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot     COMPLETE              32.26
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR             COMPLETE              32.21
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta      COMPLETE              30.91
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2        COMPLETE               7.86
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot     COMPLETE              12.26
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              14.82
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR             COMPLETE              16.82
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP              COMPLETE              42.06
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta      COMPLETE              19.26
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_RAP                 COMPLETE              10.46
grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2        COMPLETE               7.49
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16                COMPLETE              20.21
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_RRFS_v1beta            COMPLETE              15.01
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2         COMPLETE             237.95
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson  COMPLETE             325.06
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16           COMPLETE             304.69
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR              COMPLETE             377.28
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta       COMPLETE             368.07
grid_RRFS_CONUScompact_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16   COMPLETE              29.90
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_HRRR           COMPLETE              27.22
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta    COMPLETE              25.99
grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16   COMPLETE              12.08
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR          COMPLETE              27.81
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta    COMPLETE               8.55
grid_RRFS_CONUScompact_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16    COMPLETE             273.15
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_HRRR            COMPLETE             332.52
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta     COMPLETE             331.76
grid_RRFS_NA_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP                 COMPLETE              72.18
grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_WoFS_v0         COMPLETE              13.72
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_HRRR                COMPLETE              16.83
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0              COMPLETE              13.56
grid_SUBCONUS_Ind_3km_ics_NAM_lbcs_NAM_suite_GFS_v16               COMPLETE              19.05
grid_SUBCONUS_Ind_3km_ics_RAP_lbcs_RAP_suite_RRFS_v1beta_plot      COMPLETE               9.10
MET_ensemble_verification_only_vx                                  COMPLETE               1.25
MET_verification_only_vx                                           COMPLETE               0.26
nco                                                                COMPLETE               7.98
nco_ensemble                                                       COMPLETE              67.92
nco_grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16      COMPLETE              28.01
nco_grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_timeoffset_suite_  COMPLETE              11.90
nco_grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thom  COMPLETE             328.61
nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR       DEAD                   1.36
pregen_grid_orog_sfc_climo                                         COMPLETE               9.38
specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS                              COMPLETE               6.48
specify_template_filenames                                         COMPLETE               8.16
----------------------------------------------------------------------------------------------------
Total                                                              DEAD                5298.31

A rerun of the get_from_NOMADS_ics_FV3GFS_lbcs_FV3GFS test successfully passed:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
get_from_NOMADS_ics_FV3GFS_lbcs_FV3GFS                             COMPLETE              13.54
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE              13.54

However, I had to add load("wget") to the modulefiles/build_orion_intel.lua modulefile in order to allow the get_extrn_ics and get_extrn_lbcs tasks to successfully pull the necessary data from NOMADS.

Without the above modification, the getextrn* tasks will all fail with the following message in the log files:

DEBUG: Running command:
 wget -q -c -T 15 -t 2 https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20231015/00/atmos/gfs.t00z.atmanl.nc

INFO: Command 'wget -q -c -T 15 -t 2 https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20231015/00/atmos/gfs.t00z.atmanl.nc' returned non-zero exit status 5.

DEBUG: Retrieved status: False

EdwardSnyder-NOAA commented 1 year ago

I have ran the comprehensive tests on the three noaacloud platforms using the newly added comprehensive.noaacloud file. The failures were captured here. All the tests have passed with the exception of get_from_NOMADS_ics_FV3GFS_lbcs_FV3GFS on AWS and Azure. Using the suggestion of adding load("wget") to the build module file doesn't work. It appears adding the additional flag --no-check-certificate to the wget command allows the NOMADS date to be downloaded.

MichaelLueken commented 1 year ago

The comprehensive testing has completed on Gaea:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
2020_CAD                                                           COMPLETE              52.14
community                                                          COMPLETE              31.34
custom_ESGgrid                                                     COMPLETE              31.44
custom_ESGgrid_Central_Asia_3km                                    COMPLETE              38.96
custom_ESGgrid_IndianOcean_6km                                     COMPLETE              32.99
custom_ESGgrid_NewZealand_3km                                      COMPLETE              75.58
custom_ESGgrid_Peru_12km                                           COMPLETE              37.50
custom_ESGgrid_SF_1p1km                                            COMPLETE             208.59
custom_GFDLgrid__GFDLgrid_USE_NUM_CELLS_IN_FILENAMES_eq_FALSE      COMPLETE              17.84
custom_GFDLgrid                                                    COMPLETE              17.46
deactivate_tasks                                                   COMPLETE               2.18
get_from_AWS_ics_GEFS_lbcs_GEFS_fmt_grib2_2022040400_ensemble_2me  COMPLETE             852.90
get_from_NOMADS_ics_FV3GFS_lbcs_FV3GFS                             COMPLETE              21.11
grid_CONUS_25km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16      COMPLETE              18.95
grid_CONUS_3km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta   COMPLETE             312.40
grid_RRFS_AK_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot        COMPLETE             155.95
grid_RRFS_AK_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR                 COMPLETE             210.61
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP              COMPLETE              42.41
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot     COMPLETE              47.83
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR             COMPLETE              41.82
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta      COMPLETE              38.78
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2        COMPLETE               9.78
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot     COMPLETE              21.40
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              20.79
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR             COMPLETE              19.97
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP              COMPLETE              76.22
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta      COMPLETE              25.44
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_RAP                 COMPLETE              20.77
grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2        COMPLETE              15.02
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16                COMPLETE              39.38
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_RRFS_v1beta            COMPLETE              23.29
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2         COMPLETE             299.11
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson  COMPLETE             422.38
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16           COMPLETE             387.87
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR              COMPLETE             429.92
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta       COMPLETE             449.35
grid_RRFS_CONUScompact_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16   COMPLETE              40.43
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_HRRR           COMPLETE              41.10
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta    COMPLETE              35.31
grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16   COMPLETE              18.95
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR          COMPLETE              48.73
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta    COMPLETE              14.92
grid_RRFS_CONUScompact_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16    COMPLETE             337.51
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_HRRR            COMPLETE             360.77
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta     COMPLETE             383.36
grid_RRFS_NA_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP                 COMPLETE              98.74
grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_WoFS_v0         DEAD                  22.32
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_HRRR                COMPLETE              25.80
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0              COMPLETE              20.89
grid_SUBCONUS_Ind_3km_ics_NAM_lbcs_NAM_suite_GFS_v16               COMPLETE              25.91
grid_SUBCONUS_Ind_3km_ics_RAP_lbcs_RAP_suite_RRFS_v1beta_plot      COMPLETE              14.51
MET_ensemble_verification_only_vx                                  DEAD                   4.39
MET_verification_only_vx                                           DEAD                   2.26
nco                                                                COMPLETE              14.02
nco_ensemble                                                       COMPLETE             207.04
nco_grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16      COMPLETE              54.14
nco_grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_timeoffset_suite_  COMPLETE              21.47
nco_grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thom  COMPLETE             450.35
nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR       DEAD                   3.83
pregen_grid_orog_sfc_climo                                         COMPLETE              16.51
specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS                              COMPLETE              14.61
specify_template_filenames                                         COMPLETE              16.80
----------------------------------------------------------------------------------------------------
Total                                                              DEAD                6844.14

The three verification tests were rerun and successfully passed after increasing the walltime for APCP tasks:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_WoFS_v0         COMPLETE              22.37
MET_ensemble_verification_only_vx                                  COMPLETE               4.44
MET_verification_only_vx                                           COMPLETE               1.22
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE              28.03

MichaelLueken commented 1 year ago

The comprehensive tests on Derecho have completed:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
2020_CAD                                                           COMPLETE              33.10
community                                                          COMPLETE              39.69
custom_ESGgrid                                                     COMPLETE              14.93
custom_ESGgrid_Central_Asia_3km                                    DEAD                   0.60
custom_ESGgrid_IndianOcean_6km                                     COMPLETE              21.63
custom_ESGgrid_NewZealand_3km                                      DEAD                   0.70
custom_ESGgrid_Peru_12km                                           COMPLETE              21.36
custom_ESGgrid_SF_1p1km                                            COMPLETE             144.93
custom_GFDLgrid__GFDLgrid_USE_NUM_CELLS_IN_FILENAMES_eq_FALSE      COMPLETE              10.56
custom_GFDLgrid                                                    COMPLETE               9.67
deactivate_tasks                                                   COMPLETE               0.79
get_from_AWS_ics_GEFS_lbcs_GEFS_fmt_grib2_2022040400_ensemble_2me  COMPLETE             639.57
get_from_NOMADS_ics_FV3GFS_lbcs_FV3GFS                             COMPLETE              12.27
grid_CONUS_25km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16      COMPLETE              16.47
grid_CONUS_3km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta   DEAD                   1.93
grid_RRFS_AK_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot        DEAD                   0.57
grid_RRFS_AK_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR                 COMPLETE             162.57
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP              COMPLETE              31.35
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot     COMPLETE              35.42
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR             COMPLETE              31.62
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta      COMPLETE              31.41
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2        COMPLETE              12.75
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot     COMPLETE              25.64
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              26.14
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR             COMPLETE              37.70
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP              COMPLETE              73.16
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta      COMPLETE              39.22
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_RAP                 COMPLETE              17.16
grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2        COMPLETE              11.26
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16                COMPLETE              41.65
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_RRFS_v1beta            COMPLETE              33.55
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2         COMPLETE             211.76
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson  COMPLETE             299.99
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16           COMPLETE             303.22
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR              COMPLETE             321.63
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta       COMPLETE             320.96
grid_RRFS_CONUScompact_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16   COMPLETE              31.42
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_HRRR           COMPLETE              26.91
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta    COMPLETE              25.85
grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16   COMPLETE              19.88
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR          COMPLETE              30.77
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta    COMPLETE              16.74
grid_RRFS_CONUScompact_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16    COMPLETE             252.49
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_HRRR            COMPLETE             264.04
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta     COMPLETE             262.73
grid_RRFS_NA_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP                 COMPLETE              76.26
grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_WoFS_v0         COMPLETE              30.80
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_HRRR                COMPLETE              38.08
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0              COMPLETE              30.28
grid_SUBCONUS_Ind_3km_ics_NAM_lbcs_NAM_suite_GFS_v16               COMPLETE              48.86
grid_SUBCONUS_Ind_3km_ics_RAP_lbcs_RAP_suite_RRFS_v1beta_plot      COMPLETE              15.81
MET_ensemble_verification_only_vx                                  COMPLETE               0.66
MET_verification_only_vx                                           COMPLETE               0.12
nco                                                                COMPLETE              19.52
nco_ensemble                                                       COMPLETE             107.19
nco_grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16      COMPLETE              29.55
nco_grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_timeoffset_suite_  COMPLETE              22.34
nco_grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thom  COMPLETE             294.56
nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR       DEAD                   0.99
pregen_grid_orog_sfc_climo                                         COMPLETE              13.49
specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS                              COMPLETE              10.87
specify_template_filenames                                         COMPLETE              14.03
----------------------------------------------------------------------------------------------------
Total                                                              DEAD                4721.17

These results are identical to what @natalie-perlin found in her PR #894 (please this comment).

In the above failed tests (not nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR), the make_sfc_climo task is failing with the following error message:

PE 8: process_vm_readv copy size mismatch: requested: 12094344 bytes copied 8441896 iter 0 total_bytes_copied 0 src_len 12094344
Assertion failed in file ../src/mpid/ch4/shm/cray_common/cray_common_memops.c at line 470: 0

MichaelLueken commented 1 year ago

@natalie-perlin - I will go ahead and move forward with merging this PR into release/public-v2.2.0 at this time.

As @mkavulich and @gspetro-NOAA have noted, the:

data: 
   ics_lbcs:

sections are for archived data sets that might be staged on a given machine. These directories should not be the same as those found in the TEST_* directories. The use of USE_USER_STAGED_DATA: true should force the use of the data staged in the TEST_* directories. Otherwise, the workflow generation should check both the:

data: 
   ics_lbcs:

directories and the TEST_* directories before moving on to a different method of retrieving data.