RatkoVasic-NOAA
commented
6 months ago With change in PR #1041 (cmake version) and change to correct spack-stack path, SRW is running on Rocky. Still issue with some HPSS jobs:
All 74 experiments finished
Calculating core-hour usage and printing final summary
----------------------------------------------------------------------------------------------------
Experiment name | Status | Core hours used
----------------------------------------------------------------------------------------------------
2020_CAD_20240217061806 COMPLETE 33.38
community_20240217061808 COMPLETE 18.66
custom_ESGgrid_20240217061809 COMPLETE 14.17
custom_ESGgrid_Central_Asia_3km_20240217061811 COMPLETE 25.88
custom_ESGgrid_Great_Lakes_snow_8km_20240217061812 COMPLETE 12.20
custom_ESGgrid_IndianOcean_6km_20240217061814 COMPLETE 19.30
custom_ESGgrid_NewZealand_3km_20240217061815 COMPLETE 53.75
custom_ESGgrid_Peru_12km_20240217061816 COMPLETE 18.11
custom_ESGgrid_SF_1p1km_20240217061817 COMPLETE 160.90
custom_GFDLgrid__GFDLgrid_USE_NUM_CELLS_IN_FILENAMES_eq_FALSE_202 COMPLETE 7.86
custom_GFDLgrid_20240217061820 COMPLETE 7.19
deactivate_tasks_20240217061821 COMPLETE 0.78
get_from_AWS_ics_GEFS_lbcs_GEFS_fmt_grib2_2022040400_ensemble_2me COMPLETE 763.25
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2019061200_2024021 COMPLETE 6.25
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2019061200_202402 DEAD 0.81
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2021032018_202402 DEAD 2.62
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_netcdf_2022060112_48h_20 DEAD 2.38
get_from_HPSS_ics_GDAS_lbcs_GDAS_fmt_netcdf_2022040400_ensemble_2 DEAD 7.80
get_from_HPSS_ics_GSMGFS_lbcs_GSMGFS_20240217061830 COMPLETE 6.63
get_from_HPSS_ics_HRRR_lbcs_RAP_20240217061832 COMPLETE 14.35
get_from_HPSS_ics_RAP_lbcs_RAP_20240217061833 COMPLETE 17.27
get_from_NOMADS_ics_FV3GFS_lbcs_FV3GFS_20240217061834 COMPLETE 12.58
grid_CONUS_25km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_202 COMPLETE 8.36
grid_CONUS_3km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_ COMPLETE 265.68
grid_RRFS_AK_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20240 COMPLETE 140.60
grid_RRFS_AK_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_20240217061839 COMPLETE 178.33
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP_20240217061 COMPLETE 24.63
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20 COMPLETE 32.26
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_2024021706 COMPLETE 29.30
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_202 COMPLETE 29.08
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20240 COMPLETE 6.18
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20 COMPLETE 12.60
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot COMPLETE 13.09
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_2024021706 COMPLETE 15.72
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP_20240217061 COMPLETE 43.97
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_202 COMPLETE 18.18
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_RAP_20240217061854 COMPLETE 10.62
grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2_20240 COMPLETE 6.43
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2024021706185 COMPLETE 20.71
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_RRFS_v1beta_202402170 COMPLETE 15.94
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_202402 COMPLETE 236.75
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson COMPLETE 330.46
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_20240217 COMPLETE 312.57
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_20240217061 COMPLETE 379.51
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_2024 COMPLETE 374.25
grid_RRFS_CONUScompact_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_ COMPLETE 29.24
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_HRRR_20240217 COMPLETE 28.66
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2 COMPLETE 27.11
grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_ COMPLETE 10.74
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024021 COMPLETE 29.11
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2 COMPLETE 9.10
grid_RRFS_CONUScompact_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_2 COMPLETE 280.68
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_HRRR_202402170 COMPLETE 339.52
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_20 COMPLETE 333.45
grid_RRFS_NA_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP_20240217061919 COMPLETE 70.66
grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_WoFS_v0_202402 COMPLETE 14.97
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024021706192 COMPLETE 17.64
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0_20240217061 COMPLETE 14.41
grid_SUBCONUS_Ind_3km_ics_NAM_lbcs_NAM_suite_GFS_v16_202402170619 COMPLETE 20.25
grid_SUBCONUS_Ind_3km_ics_RAP_lbcs_RAP_suite_RRFS_v1beta_plot_202 COMPLETE 8.85
long_fcst_20240217061927 COMPLETE 81.95
MET_ensemble_verification_only_vx_20240217061928 COMPLETE 1.09
MET_ensemble_verification_only_vx_time_lag_20240217061932 COMPLETE 3.77
MET_ensemble_verification_winter_wx_20240217061935 COMPLETE 112.29
MET_verification_only_vx_20240217061937 COMPLETE 0.23
nco_20240217061940 COMPLETE 8.99
nco_ensemble_20240217061943 COMPLETE 61.65
nco_grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_202 COMPLETE 27.83
nco_grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_timeoffset_suite_ DEAD 1.21
nco_grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thom COMPLETE 324.99
nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR_2024 COMPLETE 11.85
pregen_grid_orog_sfc_climo_20240217061957 COMPLETE 6.73
specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS_20240217062000 COMPLETE 6.86
specify_template_filenames_20240217062001 COMPLETE 8.28
----------------------------------------------------------------------------------------------------
Total DEAD 5603.45
Detailed summary written to /scratch2/NCEPDEV/fv3-cam/Ratko.Vasic/Rocky8/expt_dirs/WE2E_summary_20240217080920.txt
Description
Hera is switching to new OS, from CentOS to Rocky. There are already 4 login nodes with new OS: hfe01-hfe08 Hera head node (CentOS) hfe09-hfe12 Hera Rocky 8 head node
Most of spack-stack libraries are already installed under /scratch1/NCEPDEV/nems/role.epic/spack-stack:
spack-stack-1.3.0/envs/unified-env-rocky8 spack-stack-1.3.1/envs/unified-env-rocky8 spack-stack-1.4.0/envs/unified-env-rocky8 spack-stack-1.4.1/envs/unified-env-rocky8 spack-stack-1.5.0/envs/unified-env-rocky8 spack-stack-1.5.1/envs/gsi-addon-env-rocky8 spack-stack-1.5.1/envs/unified-env-rocky8 spack-stack-1.6.0/envs/gsi-addon-dev-rocky8 spack-stack-1.6.0/envs/unified-env-rocky8
Solution
Login to one of hfe09-hfe12 nodes, checkout SRW app, change paths in build_hera_gnu.lua orbuild_hera_intel.lua build run some tests report any bugs
Acceptance Criteria (Definition of Done)
Once it pass tests.