WalterKolczynski-NOAA commented 1 month ago

Description

The workflow was updated to use modules built on Rocky 9, but the external packages (like prepobs) were still pointing to the versions built on CentOS (Orion). This transitions to packages built on Rocky 9.

Updating of the tracker package has been deferred until later. As such, the tracker jobs have been disabled by returning immediately if they are on Hercules. Since these jobs are small, resource-wise, it should not meaningfully impact turnover time.

Type of change

New feature (adds functionality)

Change characteristics

Is this a breaking change (a change in existing functionality)? YES
- Tracker has not yet been updated to build on Hercules
Does this change require a documentation update? NO

How has this been tested?

Cycled test on Hercules

Checklist

[x] Any dependent changes have been merged and published
[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] My changes generate no new warnings
[ ] New and existing tests pass with my changes
[x] I have made corresponding changes to the documentation if necessary

WalterKolczynski-NOAA commented 1 month ago

The only update needed to the packages was a minor change to the Fit2Obs spack-stack module path:

-prepend_path("MODULEPATH", "/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/unified-env/install/modulefiles/Core")
+prepend_path("MODULEPATH", "/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/gsi-addon-env/install/modulefiles/Core")

Changes are made in-place without changing the version number of the directory (yet).

emcbot commented 1 month ago

Experiment C48_ATM FAILED on Hercules with error logs:

/work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/COMROOT/C48_ATM_1f1ac3e4/logs/2021032312/gfsgenesis.log
/work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/COMROOT/C48_ATM_1f1ac3e4/logs/2021032312/gfstracker.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 1 month ago

Experiment C48_S2SW FAILED on Hercules in /work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/C48_S2SW_1f1ac3e4

emcbot commented 1 month ago

Experiment C48_ATM FAILED on Hercules in /work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/C48_ATM_1f1ac3e4

emcbot commented 1 month ago

Experiment C96_atm3DVar FAILED on Hercules with error logs:

/work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/COMROOT/C96_atm3DVar_1f1ac3e4/logs/2021122100/gfsgenesis.log
/work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/COMROOT/C96_atm3DVar_1f1ac3e4/logs/2021122100/gfstracker.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 1 month ago

Experiment C96C48_hybatmDA FAILED on Hercules in /work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/C96C48_hybatmDA_1f1ac3e4

emcbot commented 1 month ago

Experiment C96_atm3DVar FAILED on Hercules in /work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/C96_atm3DVar_1f1ac3e4

DavidHuber-NOAA commented 1 month ago

It looks like the TC_Tracker is not in the specified location:

hercules-login-1[3] ~$ ls /work/noaa/global/glopara/git_rocky9/TC_tracker/
hercules-login-1[4] ~$

WalterKolczynski-NOAA commented 1 month ago

It looks like the TC_Tracker is not in the specified location:
hercules-login-1[3] ~$ ls /work/noaa/global/glopara/git_rocky9/TC_tracker/
hercules-login-1[4] ~$

Yeah, this was a known thing. I thought I had tested this before I launched CI, but might've mixed it up with another test I did over the weekend.

emcbot commented 1 month ago

Experiment C48_ATM FAILED on Hercules with error logs:

/work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/COMROOT/C48_ATM_ecac7485/logs/2021032312/gfsgenesis.log
/work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/COMROOT/C48_ATM_ecac7485/logs/2021032312/gfstracker.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 1 month ago

Experiment C48_S2SW FAILED on Hercules in /work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/C48_S2SW_ecac7485

emcbot commented 1 month ago

Experiment C48_ATM FAILED on Hercules in /work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/C48_ATM_ecac7485

emcbot commented 1 month ago

Experiment C96_atm3DVar FAILED on Hercules in /work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/C96_atm3DVar_ecac7485

emcbot commented 1 month ago

Experiment C96C48_hybatmDA FAILED on Hercules in /work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/C96C48_hybatmDA_ecac7485

emcbot commented 1 month ago

Experiment C48_S2SWA_gefs FAILED on Hercules in /work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/C48_S2SWA_gefs_ecac7485

emcbot commented 1 month ago

Experiment C48_ATM FAILED on Hercules with error logs:

/work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/COMROOT/C48_ATM_94b5af9c/logs/2021032312/gfsgenesis.log
/work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/COMROOT/C48_ATM_94b5af9c/logs/2021032312/gfstracker.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 1 month ago

Experiment C48_S2SW FAILED on Hercules in /work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/C48_S2SW_94b5af9c

emcbot commented 1 month ago

Experiment C48_ATM FAILED on Hercules in /work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/C48_ATM_94b5af9c

emcbot commented 1 month ago

Experiment C96C48_hybatmDA FAILED on Hercules with error logs:

/work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/COMROOT/C96C48_hybatmDA_94b5af9c/logs/2021122100/gfsgenesis.log
/work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/COMROOT/C96C48_hybatmDA_94b5af9c/logs/2021122100/gfstracker.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 1 month ago

Experiment C96C48_hybatmDA FAILED on Hercules in /work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/C96C48_hybatmDA_94b5af9c

emcbot commented 1 month ago

Experiment C48_S2SWA_gefs FAILED on Hercules in /work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/C48_S2SWA_gefs_94b5af9c

emcbot commented 1 month ago

Experiment C96_atm3DVar FAILED on Hercules in /work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/C96_atm3DVar_94b5af9c

emcbot commented 1 month ago

CI Passed Hercules at
Built and ran in directory /work2/noaa/stmp/CI/HERCULES/2608

emcbot commented 1 month ago


CI Update on Wcoss2 at 05/30/24 02:08:17 PM
============================================
Cloning and Building global-workflow PR: 2608
with PID: 55992 on host: clogin01

emcbot commented 1 month ago

Automated global-workflow Testing Results:


Machine: Wcoss2
Start: Thu May 30 14:17:49 UTC 2024 on clogin01
---------------------------------------------------
Build: Completed at 05/30/24 02:30:19 PM
Case setup: Completed for experiment C48_ATM_d45b7ca5
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_d45b7ca5
Case setup: Skipped for experiment C48_S2SWA_gefs_d45b7ca5
Case setup: Completed for experiment C48_S2SW_d45b7ca5
Case setup: Completed for experiment C96_atm3DVar_extended_d45b7ca5
Case setup: Skipped for experiment C96_atm3DVar_d45b7ca5
Case setup: Skipped for experiment C96_atmaerosnowDA_d45b7ca5
Case setup: Completed for experiment C96C48_hybatmDA_d45b7ca5
Case setup: Skipped for experiment C96C48_ufs_hybatmDA_d45b7ca5

emcbot commented 1 month ago

Experiment C48_S2SW FAILED on Hera with error logs:

/scratch1/NCEPDEV/global/CI/2608/RUNTESTS/COMROOT/C48_S2SW_d45b7ca5/logs/2021032312/gfswaveinit.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 1 month ago

Experiment C96_atm3DVar FAILED on Hera in /scratch1/NCEPDEV/global/CI/2608/RUNTESTS/C96_atm3DVar_d45b7ca5

emcbot commented 1 month ago

Experiment C96_atmaerosnowDA FAILED on Hera in /scratch1/NCEPDEV/global/CI/2608/RUNTESTS/C96_atmaerosnowDA_d45b7ca5

emcbot commented 1 month ago

Experiment C48mx500_3DVarAOWCDA FAILED on Hera in /scratch1/NCEPDEV/global/CI/2608/RUNTESTS/C48mx500_3DVarAOWCDA_d45b7ca5

emcbot commented 1 month ago

Experiment C96C48_hybatmDA FAILED on Hera in /scratch1/NCEPDEV/global/CI/2608/RUNTESTS/C96C48_hybatmDA_d45b7ca5

emcbot commented 1 month ago

Experiment C48_S2SW FAILED on Hera in /scratch1/NCEPDEV/global/CI/2608/RUNTESTS/C48_S2SW_d45b7ca5

emcbot commented 1 month ago

Experiment C48_ATM FAILED on Hera with error logs:

/scratch1/NCEPDEV/global/CI/2608/RUNTESTS/COMROOT/C48_ATM_d45b7ca5/logs/2021032312/gfsfcst.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 1 month ago

Experiment C48_S2SWA_gefs FAILED on Hera in /scratch1/NCEPDEV/global/CI/2608/RUNTESTS/C48_S2SWA_gefs_d45b7ca5

emcbot commented 1 month ago

Experiment C48_ATM FAILED on Hera in /scratch1/NCEPDEV/global/CI/2608/RUNTESTS/C48_ATM_d45b7ca5

emcbot commented 1 month ago

CI Passed Orion at
Built and ran in directory /work2/noaa/stmp/CI/ORION/2608

emcbot commented 1 month ago

Experiment C48_ATM_d45b7ca5 SUCCESS on Wcoss2 at 05/30/24 06:27:15 PM

emcbot commented 1 month ago

Experiment C48_S2SW_d45b7ca5 SUCCESS on Wcoss2 at 05/30/24 06:27:19 PM

emcbot commented 1 month ago

Experiment C96C48_hybatmDA_d45b7ca5 SUCCESS on Wcoss2 at 05/30/24 06:42:22 PM

emcbot commented 1 month ago

Experiment C96_atm3DVar_extended_d45b7ca5 SUCCESS on Wcoss2 at 05/31/24 06:33:26 AM

emcbot commented 1 month ago

All CI Test Cases Passed on Wcoss2:


Experiment C48_ATM_d45b7ca5 *** SUCCESS *** at 05/30/24 06:27:15 PM
Experiment C48_S2SW_d45b7ca5 *** SUCCESS *** at 05/30/24 06:27:19 PM
Experiment C96C48_hybatmDA_d45b7ca5 *** SUCCESS *** at 05/30/24 06:42:22 PM
Experiment C96_atm3DVar_extended_d45b7ca5 *** SUCCESS *** at 05/31/24 06:33:26 AM

emcbot commented 1 month ago

CI Passed Hera at
Built and ran in directory /scratch1/NCEPDEV/global/CI/2608

NOAA-EMC / global-workflow

Switch to Rocky 9 built external packages on Hercules #2608

Description

Type of change

Change characteristics

How has this been tested?

Checklist